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Preface 


If  managed  wisely,  investments  in  information  technology  (IT)  can  enrich 
people's  lives  and  improve  organizational  performance.  For  example, 
during  the  last  decade  the  Internet  has  matured  from  being  a  technical 
novelty  to  a  national  resource  where  citizens  can  visit  the  Library  of 
Congress  or  file  their  tax  returns.  Some  organizations  have  realized 
substantial  improvements  in  processing  data  and  information  by  switching 
from  centralized  mainframe  computing  to  decentralized  personal 
computers  linked  by  local  area  networks.  The  ability  of  software 
applications  to  locate  and  correlate  relevant  data  in  a  data  warehouse 
permits  organizations  to  discover  unknown  fiscal  or  physical  resource 
relationships  and  thus  provide  appropriate  assistance  where  there  had 
been  none. 

However,  along  with  the  potential  to  improve  lives  and  organizations,  IT 
projects  can  become  risky,  costly,  unproductive  mistakes.  As  we  have 
described  in  numerous  reports  and  testimonies,  federal  IT  projects  too 
frequently  incur  cost  overruns  and  schedule  slippages  while  contributing 
little  to  mission-related  outcomes. 

The  Clinger-Cohen  Act  of  19961  was  enacted  to  address  many  of  the 
problems  related  to  federal  IT  management.  It  requires  federal  agencies  to 
focus  more  on  the  results  achieved  through  IT  investments  while 
concurrently  streamlining  the  IT  acquisition  process.  This  act  also 
introduced  more  rigor  and  structure  into  how  agencies  select  and  manage 
IT  projects.  Among  other  things,  the  head  of  each  agency  is  required  to 
implement  a  process  for  maximizing  the  value  of  the  agency's  IT 
investments  and  assessing  and  managing  the  risks  of  its  IT  acquisitions. 

In  1997  we  developed  guidance,  based  primarily  on  the  Clinger-Cohen  Act, 
that  provides  a  method  for  evaluating  and  assessing  how  well  a  federal 
agency  is  selecting  and  managing  its  IT  resources  and  identifies  specific 
areas  where  improvements  can  be  made.  The  Information  Technology 
Investment  Management  (ITIM)  framework  enhances  this  guidance  by 
identifying  critical  processes  for  successful  IT  investment  and  organizing 
these  processes  into  a  framework  of  increasingly  mature  stages.  This  shift 
reflects  both  the  maturation  of  the  thinking  in  the  area  of  IT  investment 
management  and  the  feedback  we  received  from  organizations  based  upon 
their  experiences  creating  their  IT  investment  mechanisms  and  processes. 
Such  a  maturity  framework  can  be  used  to  analyze  an  organization's  IT 


The  fiscal  year  1997  Omnibus  Consolidated  Appropriations  Act,  Pub.  L.  104-208,  renamed  both 
Division  D  (the  Federal  Acquisition  Reform  Act)  and  E  (the  Information  Technology 
Management  Reform  Act)  of  the  1996  DOD  Authorization  Act,  Pub.  L.  104-106,  as  the  Clinger- 
Cohen  Act  of  1996. 
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investment  management  process  and  determine  the  maturity  of  its 
investment  process.  In  doing  so,  HIM  establishes  three  key  benefits: 

(1)  a  rigorous,  standardized  tool  for  internal  and  external  evaluations  of  an 
agency's  IT  investment  management  process;  (2)  a  consistent  and 
understandable  mechanism  for  reporting  the  results  of  these  assessments 
to  agency  executives,  the  Congress,  and  other  interested  parties;  and 
(3)  a  road  map  agencies  can  use  for  improving  their  IT  investment 
management  process. 

It  should  be  noted,  however,  that  the  achievement  of  more  mature  IT 
investment  management  stages  depends  on  performing  other  good 
management  practices  and  attributes  such  as  human  capital,  training,  IT 
architecture,  and  software  management. 

The  Information  Technology  Management  Policies  Group  developed  this 
guide  under  the  direction  of  Dave  McClure,  Associate  Director, 
Governmentwide  and  Defense  Information  Systems.  Accompanying  this 
document  is  an  overview  document,  Information  Technology  Investment 
Management: An  Overview  of  GAO’s Assessment F ramework  (AIMD-00- 
155),  that  provides  a  brief  description  of  ITIM.  If  you  have  any  questions 
about  the  I  nformation  T echnology  I  nvestment  M  anagement  framework  or 
the  IT  investment  management  approach,  please  contact  J  ohn  T.  Christian, 
Senior  Business  Process  Analyst,  at  (202)  512-6205 
( christianj.aimd@gao.gov ),  or  John  P.  Rehberger,  Senior  Information 
Systems  Analyst,  at  (202)  512-3687  ( rehbergerj.aimd@gao.gov ). 

An  electronic  version  of  this  guide  is  available  from  GAO's  World  Wide 
Web  server  at  the  following  address: 

http://www.gao.gov/special.pubs/10_l_23.pdf.  Additional  copies  of  this 
exposure  draft  can  be  obtained  from  Room  1100,  700  4th  St.  N  .W.,  U  .S. 
General  Accounting  Office,  Washington,  D.C.  20548,  or  by  calling  (202) 
512-6000,  or  TDD  (202)  512-2537.  Please  send  comments  by  September  1, 
2000,  to  J  ohn  T.  Christian,  at 

U.S.  General  Accounting  Office 
441  G.  Street,  N.W.  Room  4T21 
Washington,  D.C.  20548 


J  effrey  C.  Stein hoff 

Assistant  Comptroller  General 

Accounting  and  Information  Management  Division 
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Introduction 


The  select/control/evaluate  model  has  become  a  central  tenet  of  the 
federal  IT  investment  management  approach.  The  model  was  initially 
identified  in  our  Strategic  Information  Management  (SIM)  Executive 
Guide,2  expanded  in  the  Office  of  Management  and  Budget's  IT  investment 
guidance,3  and  then  refined  in  our  subsequent  guidance.4  It  provides  a 
systematic  method  for  agencies  to  minimize  risks  while  maximizing  the 
returns  of  IT  investments.  Figure  1  illustrates  the  central  components  of 
this  model. 


Figure  1 :  Fundamental  Phases  of  the  IT  Investment  Approach 


Are  the  systems 
delivering  what 
you  expected? 


•  During  the  selection  phase  the  organization  (1)  selects  those  IT  projects 
that  will  best  support  its  mission  needs  and  (2)  identifies  and  analyzes 


Executive  Guide:  improving  Mission  Performance  Through  Strategic  Information  Management  and 
Technology  (GAO/AIMD-94-115,  May  1994). 

^Evaluating  Information  Technology  Investments,  A  Practical  Guide,  Executive  Office  of  the 
President,  Office  of  Management  and  Budget,  November  1995. 

4 Assessing  Risks  and  Returns:  A  Guide  for  Evaluating  Federal  Agencies'  IT  Investment  Decision¬ 
making  (GAO/AIM D-10.1.13,  February  1997). 
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each  project's  risks  and  returns  before  committing  significant  funds  to  a 
project. 

•  During  the  control  phase  the  organization  ensures  that,  as  projects 
develop  and  as  investment  costs  rise,  the  project  is  continuing  to  meet 
mission  needs  at  the  expected  levels  of  cost  and  risk.  If  the  project  is  not 
meeting  expectations  or  if  problems  have  arisen,  steps  are  quickly  taken  to 
address  the  deficiencies. 

•  Lastly,  during  the  evaluation  phase,  actual  versus  expected  results  are 
compared  once  projects  have  been  fully  implemented.  This  is  done  to 
(1)  assess  the  project's  impact  on  mission  performance,  (2)  identify  any 
changes  or  modifications  to  the  project  that  may  be  needed,  and  (3)  revise 
the  investment  management  process  based  on  lessons  learned. 

The  select/control/evaluate  model  presented  in  the  SIM  executive  guide 
also  provides  the  key  foundation  for  our  IT  investment  decision-making 
assessment  guide.5  That  assessment  guide  was  developed  to  provide  a 
method  for  evaluating  and  assessing  how  well  a  federal  agency  selects  and 
manages  its  IT  resources  and  to  identify  specific  areas  where 
improvements  can  be  made.  As  such,  it  expands  upon  the 
select/control/evaluate  process  model  to  incorporate  organizational 
process,  supporting  data,  and  relevant  executive  decisions. 

The  assessment  guide  is  being  used  by  agencies  and  management 
consulting  firms  to  design  and  implement  IT  investment  processes  and  by 
our  evaluators  to  assess  these  processes.  These  experiences  have 
identified  strengths  and  some  opportunities  for  improvement  for  this 
guide.  For  example,  the  comprehensive  list  of  assessment  questions 
contained  in  the  guide  thoroughly  covers  IT  investment  management 
issues.  These  questions  help  evaluators  determine  the  presence  or  absence 
of  IT  investment  process  activities.  Users  of  the  guide,  however,  expressed 
an  interest  in  a  prioritization  of  the  relative  importance  of  the  different 
process  components.  This  can  become  a  significant  issue  because 
(1)  many  agencies  must  prioritize  the  use  of  their  limited  resources  for 
improving  their  internal  processes  and  (2)  improvements  in  some  specific 
processes  can  provide  greater  benefits  to  an  organization  than 
improvements  in  other  processes. 

Additionally,  users  of  the  guide  expressed  an  interest  in  a  tool  that  would 
assist  them  in  measuring  the  interim  stages  of  development  while  the 


5  Assessing  Risks  and  Returns:  A  Guide  for  Evaluating  Federal  Agencies'  IT  Investment  Decision¬ 
making  (GAO/AIM D-10.1.13,  February  1997). 
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agency  is  implementing  a  complete  IT  investment  management  process. 
Our  evaluations  of  the  investment  management  processes  in  the  private 
sector  and  at  several  federal  agencies  indicate  that  IT  investment 
management  implementation  is  a  step-by-step  process  that  occurs  over 
time  and  depends  heavily  on  organizational  commitment,  leadership, 
persistence,  and  management  priority. 


A  Maturity  Framework 
Offers  Benefits  for 
Refining  the  IT 
Investment  Approach 


T o  address  the  issues  described  above,  we  searched  for  an  approach  that 
would  enhance  the  current  investment  management  guidance.  We  decided 
to  use  a  maturity  framework  because 

it  offers  a  comprehensive  model  for  assessing  processes  within  an 
organization,  including  engineering,  management,  and  organizational 
processes; 


•  it  can  be  applied  to  multiple  types  of  disciplines,  such  as  IT  asset 
acquisition,  human  capital,  and  systems  engineering; 


•  maturity  models  have  been  proven  to  be  a  highly  effective  evaluative 
technique  for  the  Software  Engineering  Institute,  which  is  highly  regarded 
for  its  collection  of  Capability  Maturity  Models*  (e.g.,  Capability  Maturity 
Model  for  Software?); 


•  a  maturity  framework  can  serve  as  a  valuable  tool  for  organizations  to 
improve  their  technical  development  and  management  processes;  and 

•  other  researchers  have  also  proposed  similar  IT  management  maturity 
model  approaches.7 

For  further  information  on  the  development  of  ITIM,  please  refer  to 
appendix  I 


5M Capabil ity  Maturity  Model  is  a  service  mark  of  Carnegie  Mellon  University. 

6M.  Paulk  et.  al„  Capability  Maturity  Model  for  Software  (Version  1.1),  SEI-93-TR-024. 

7G iga  Information  Group,  Inc.,  Total  Economic  Impact,  Part 2:  Defining  and  Measuring  IT 
Value  { P -1297-009). 
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Overview  of  HIM 


The  ITIM  Maturity 
Stages 


HIM  is  comprised  of  five  stages  of  maturity.  Each  stage  builds  upon  the 
lower  stages  and  enhances  the  organization's  ability  to  manage  its  IT 
investments.  Figure  2.1  shows  the  five  ITIM  stages  and  a  brief  description 
of  each  stage. 


Figure  2.1 :  The  Five  Stages  of  Maturity  Within  ITIM 


Enterprise 
and  Strategic 
Focus 


Maturity  Stages 


Project- 

Centric 


Description 


Investment  benchmarking  and  IT-enabled 
change  management  techniques  are  deployed 
to  strategically  shape  business  outcomes. 


Process  evaluation  techniques  focus  on 
improving  the  performance  and  management 
of  the  organization's  IT  investment  portfolio. 


Comprehensive  IT  investment  portfolio  selection 
and  control  techniques  are  in  place  that 
incorporate  benefit  and  risk  criteria  linked  to 
mission  goals  and  strategies. 


Repeatable  investment  control  techniques  are  in 
place  and  the  key  foundation  capabilities  have 
been  implemented. 


There  is  little  awareness  of  investment 
management  techniques.  IT  management 
processes  are  ad  hoc,  project-centric,  and 
have  widely  variable  outcomes. 


The  following  paragraphs  provide  a  more  detailed  description  of  the 
general  characteristics  and  practices  found  at  each  stage  of  maturity. 


ITIM  Stage  1:  Creating 
I  nvestment  Awareness 


Stage  1  is  characterized  by  ad  hoc,  unstructured,  and  unpredictable 
investment  processes.  For  example,  in  a  Stage  1  organization,  there  is 
generally  little  relationship  between  the  success  or  failure  of  one  project 
and  the  success  or  failure  of  another  project.  If  an  IT  project  succeeds  and 
is  seen  as  a  good  investment,  it  is  largely  due  to  exceptional  actions  on  the 
part  of  the  project  team  members  and  thus  its  success  might  be  difficult  to 
repeat.  Investment  and  development  processes  that  are  important  for 
success  may  be  known,  but  only  to  isolated  teams;  this  process  knowledge 
is  not  widely  shared  or  institutionalized. 
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The  unpredictable  nature  of  project  outcomes  means  that  even  if  an 
organization  does  recognize  that  a  given  project  is  in  trouble,  the 
organization  has  only  a  limited  ability  to  address  and  resolve  the  project's 
problems.  Additionally,  a  focus  on  project  results  in  terms  of  business 
benefits  is  often  missing  in  Stage  1  organizations. 

Most  organizations  with  Stage  1  maturity  have  some  type  of  project 
selection  process  in  place  as  part  of  their  annual  budgeting  activity. 
However,  the  selection  process  is  frequently  rudimentary,  poorly 
documented,  and  at  times  inconsistent.  Organizations,  when  evaluated 
using  ITIM,  are  assumed  to  initially  have  Stage  1  investment  maturity. 


ITIM  Stage  2:  Building  the 
Investment  Foundation 


The  primary  focus  of  Stage  2  maturity  is  on  attaining  repeatable, 
successful  IT  project-level  investment  control  processes  and  basic 
selection  processes.  For  an  organization  to  develop  an  overall  sound  IT 
investment  process,  it  must  first  be  able  to  control  its  investments  so  that 
they  finish  predictably  within  established  schedule  and  budget  ranges.  In 
the  absence  of  predictable  and  repeatable  investment  control  processes, 
selected  investments  will  be  subjected  to  a  higher  risk  of  failure  despite 
rigorous  analysis  of  the  estimates  used  to  justify  them.  F  urther,  the 
absence  of  repeatable  control  processes  will  result  in  ineffective 
evaluation  processes  and  contradictory  process  improvement  efforts. 


Most  IT  investments  require  a  relentless  focus  on  interim  results  and 
successful  risk  management  strategies  to  ultimately  succeed.  As  such,  an 
organization  can  begin  by  (1)  focusing  on  gaining  control  of  its  existing 
collection  of  projects  and  (2)  following  a  disciplined  process  for  regularly 
tracking  and  overseeing  each  project's  cost  and  schedule  milestones  and 
improving  project  outcomes  over  time.  Supporting  these  activities  requires 
the  creation  of  an  IT  asset  inventory  to  ensure  that  the  organization  knows 
certain  basic  information  about  its  IT  assets  such  as  the  location,  cost,  and 
ownership. 


Stage  2  selection-related  processes  are  designed  to  establish  basic 
selection  capabilities  that  can  evolve  into  more  mature  selection 
capabilities  in  Stage  3.  Therefore,  the  organization  also  focuses  on 
defining  and  developing  its  IT  investment  board(s),  identifying  the 
business  needs  or  opportunities  to  be  addressed  by  each  IT  project,  and 
using  this  knowledge  in  the  selection  of  new  IT  proposals. 
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ITIM  Stage  3:  Developing  a 
Complete  Investment 
Portfolio 


Establishing  a  consistent,  well-defined  IT  investment  portfolio  perspective 
is  the  critical  focus  for  Stage  3  maturation  along  with  maintaining  mature 
control  processes  and  initiating  basic  evaluation  processes.  Once  new  IT 
proposals  can  be  selected  and  developed  on  schedule  and  on  budget  per 
Stage  2,  the  organization  needs  to  consider  criteria  for  how  it  should 
develop  an  IT  investment  portfolio.  An  IT  investment  portfolio  is  not  just  a 
collection  of  projects  but  a  conscious,  proactive  look  at  how  the 
organization  expends  its  limited  resources  on  IT,  what  beneficial  impacts 
these  investments  have  on  the  organization,  and  a  continuous  search  for 
investments  that  will  better  achieve  the  organization's  mission. 


Defining  IT  investment  portfolio  selection  criteria  (1)  enables  the 
organization  to  widen  its  criteria  from  primarily  cost  and  schedule  to 
include  benefit  and  risk  criteria  and  (2)  communicates  organizational 
priorities  to  the  IT  project  management  community.  Investment  analysis 
efforts  focus  on  ensuring  that  each  investment  submitted  for  funding 
supports  the  organization's  missions,  strategies,  and  goals.  Portfolio 
development  actions  define  the  criteria  and  tasks  needed  to  develop  an  IT 
investment  portfolio.  Finally,  organizations  with  multiple  IT  investment 
boards  must  work  to  align  the  authority  of  these  multiple  IT  investment 
boards  and  describe  practices  for  supporting  such  a  management 
structure. 


ITIM  Stage  4:  Improving 
the  Investment  Process 


An  organization  at  Stage  4  maturity  is  focused  on  using  evaluation 
techniques  to  improve  its  IT  investment  processes  and  portfolio  along  with 
maintaining  mature  control  and  selection  processes.  A  key  tool  for 
accomplishing  this  is  the  post-implementation  review  (PIR).  The  PIR  is 
conducted  after  an  investment  is  completed  and  examines  the  outcome  of 
the  investment  relative  to  its  plans  and  expectations.  This  examination 
typically  identifies  lessons  learned  from  the  investment  and  improves  the 
understanding  of  the  key  variables  in  the  investment's  business  case. 
Analyzing  a  number  of  PIRs  serves  as  the  basis  for  creating 
recommendations  for  changing  and  improving  the  IT  investment 
processes. 

Portfolio  categories  are  used  to  organize  the  lessons  learned  and 
recommendations  gleaned  from  PIRs  and  other  sources  of  process  or 
investment  information.  The  information  within  these  categories  is  then 
used  to  fine-tune  the  investment  processes  and  portfolio.  Additionally,  at 
Stage  4  maturity  the  organization  has  the  capacity  to  conduct  IT 
succession  actions  and  thus  can  plan  and  implement  the  "de-selection"  of 
obsolete,  high-risk,  or  low-value  IT  investments. 
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ITIM  Stage  5:  Leveraging 
Information  Technology 
for  Strategic  Outcomes 


Once  an  organization  masters  the  selection,  control,  and  evaluation 
processes,  it  seeks  to  shape  its  strategic  outcomes  by  (1)  learning  from 
other  organizations  and  (2)  continuously  improving  the  manner  in  which  it 
uses  IT  to  support  and  improve  its  business  outcomes.  Thus,  an 
organization  with  Stage  5  maturity  benchmarks  its  IT  investment 
processes  relative  to  other  "best-in-class"  organizations  and  conducts 
proactive  monitoring  for  breakthrough  information  technologies  that  will 
allow  it  to  significantly  change  and  improve  its  business  performance. 


Progressing  Through 
the  ITIM  Stages  of 
Maturity 


Within  ITIM,  lower  maturity  stages  provide  the  foundation  for  upper 
maturity  stages.  Thus,  an  organization  increases  its  IT  investment 
maturity  and  management  capability  as  it  progresses  through  the  ITIM 
maturity  stages.  The  following  section  describes  the  critical  maturation 
steps  that  occur  as  an  organization  moves  from  one  stage  to  the  next  (see 
figure  2.2). 


Figure  2.2:  ITIM  Stages  of  Maturity  and  Critical  Maturation  Steps 


Maturity  Stages 


Stage  5 

Leveraging  IT  for 
Strategic  Outcomes 


Stage  4 

Improving  the 
Investment  Process 


Stage  3 

Developing  a  Complete 
Investment  Portfolio 


Stage  2 

Building  the 
Investment  Foundation 


Stage  1 

Creating  Investment 
Awareness 


Critical  Maturation  Steps 

•  Focus  on  improving  strategic  outcomes 

•  Capability  to  change  business  processes  to 
take  advantage  of  technology  changes 

•  Learn  from  others  by  benchmarking 

^processes _ 


•  Development  of  mature  evaluation  processes 

•  Capability  to  modify  IT  investment 

management  process  resulting  in  more 
favorable  outcomes _ 

"  > 

•  Development  of  mature  selection  processes 

•  Movement  from  project-based  to  portfolio- 
based  IT  management 

•  Collection  of  cost,  benefit,  schedule,  and 
risk  data  for  all  projects 

- - y 

'  N 

•  Better  understanding  the  IT  investment 
approach 

•  Development  of  mature  control  processes 

•  Maintenance  of  basic  selection  processes  , 


Moving  From  Stage  1  to 
Stage  2 


Investment  control  processes  are  the  essential  proficiencies  established  by 
an  organization  as  it  moves  from  ITI M  Stage  1  to  Stage  2.  As  investment 
control  processes  become  better  established; 
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•  one  or  more  IT  investment  board(s)  is  created  to  oversee  and  select  IT 
projects; 

•  an  IT  asset  inventory  is  created  to  support  executive  decision-making; 

•  visibility  into  IT  projects  (from  an  investment  perspective)  increases; 

•  ongoing  projects  more  predictably  achieve  their  interim  and  final 
development  and  schedule  milestones  because  of  improved 
organizationwide  system  acquisition,  development,  and  management 
practices; 

•  the  organization  creates  and  maintains  better  project-level  cost 
accountability;  and 

•  key  customers  (or  end  users)  and  business  needs  for  each  IT  project  are 
identified. 

Critical  to  maturing  project-level  IT  investment  control  processes  is  the 
ability  to  recognize  the  need  for  and  to  take  swift  corrective  action  when  a 
project  is  having  trouble  meeting  its  schedule  expectations  and  cost 
estimates.  As  the  organization  matures,  it  learns  from  past  decisions, 
better  manages  the  causal  factors  that  created  the  past  problems,  and  thus 
improves  the  cost  and  schedule  results  in  ongoing  projects. 

Beyond  the  investment  control  processes,  the  organization  also  begins  to 
implement  basic  selection  processes.  The  core  business  needs  for  each  IT 
project  are  identified  and  the  basic  portfolio  development  processes  are 
used  to  select  new  IT  proposals. 


Moving  From  Stage  2  to 
Stage  3 


Creation  of  a  mature  IT  investment  selection  process  is  the  major 
accomplishment  demonstrated  as  an  organization  moves  from  Stage  2  to 
Stage  3  maturity.  Well-developed  investment  control  processes  lead  to 
greater  certainty  about  future  IT  investment  outcomes  and  greater 
confidence  that  IT  investments,  when  they  are  selected,  will  achieve  their 
expected  cost  and  schedule  goals.  Thus,  once  the  investment  control 
processes  have  been  established,  an  organization  can  build  mature 
portfolio  selection  processes.  Mature  selection  processes  include 

the  creation  and  maintenance  of  portfolio  selection  criteria, 

the  analysis  associated  with  examining  the  merits  of  each  IT  investment, 
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the  grouping  of  similar  investments  together  and  the  development  of  the 
portfolio,  and 

the  creation  of  a  mechanism  to  coordinate  multiple  IT  investment  boards 
(if  multiple  boards  exist). 

Beyond  the  creation  of  a  mature  selection  process,  the  organization  now 
adds  the  elements  of  benefit  and  risk  management  to  its  investment 
control  process  since  it  has  installed  the  supporting  tools  for  doing  so  as 
part  of  selection  process  maturation. 


Moving  From  Stage  3  to 
Stage  4 


As  an  organization  reaches  Stage  4  maturity,  it  has  created  mature  IT 
investment  evaluation  processes  and  established  a  complete  IT  investment 
management  process.  I  n  this  stable  environment,  the  organization  can 
take  the  lessons  it  has  learned  from  evaluating  its  investment  processes 
(i.e.,  based  on  post-implementation  reviews)  and  change  these  processes 
with  predictably  beneficial  results.  By  doing  so,  it  also  creates  the 
environment  and  the  mechanisms  for  continuous  improvement  in  Stage  5. 

I  n  addition  to  investment  process  improvement,  the  organization  can  also 
manage  resource  succession-that  is,  "de-selecting"  current  IT  investments 
by  migrating  to  successor  IT  investments  or  retiring  obsolete  and  low- 
performing  IT  investments. 


Moving  From  Stage  4 to 
Stage  5 


An  organization  that  is  maturing  from  Stage  4  to  Stage  5  has  mature 
selection,  control,  and  evaluation  processes  in  place.  The  organization 
now  seeks  ways  to  (1)  institutionalize  the  continuous  improvement  of 
these  processes  and  (2)  improve  its  strategic  business  outcomes.  It 
accomplishes  these  goals  by  examining  and  learning  from  others  by  means 
of  benchmarking.  Benchmarking  is  used  by  the  organization  because  there 
may  be  external  organizations  that  have  specific  processes  that  are  more 
innovative  or  more  efficient  than  its  own  processes.  Beyond 
benchmarking,  the  organization  leverages  IT  to  significantly  change  and 
improve  its  business  performance  and  outcomes. 
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ITIM  Hierarchy 

Like  other  maturity  models,  ITIM  is  subdivided  into  a  hierarchy.  Thus, 

ITIM  is  characterized  by  subdividing  the  IT  investment  management 
process  into  five  maturity  stages.  Each  maturity  stage  consists  of 

critical  processes  that  are  defined  by  core  elements.  Each  core 
element  is  composed  of  a  number  of  key  practices.  These  hierarchical 
components  are  described  below. 

Maturity  Stages 

Each  of  the  four  maturity  stages  beyond  Stage  1  is  a  plateau  of  well- 
defined  critical  processes.  The  five  maturity  stages  represent  the  steps 
toward  achieving  a  mature,  comprehensive  IT  investment  management 
process. 

Critical  Processes 

With  the  exception  of  Stage  1,  each  maturity  stage  is  composed  of  multiple 
critical  processes,  such  as  the  processes  used  to  create  an  IT  investment 
portfolio.  Each  critical  process  contains  a  set  of  common  attributes- its 
core  elements-that  when  fulfilled,  implement  the  critical  process  needed 
to  attain  a  given  maturity  stage. 

Core  Elements 

The  core  elements  provide  the  common  framework  for  each  critical 
process.  The  five  types  of  core  elements  (purpose,  organizational 
commitment,  prerequisites,  activities,  and  evidence  of  performance),  their 
relationship  to  each  other,  and  an  explanation  of  each  core  element  are 
presented  in  figure  3.1. 

Key  Practices 

The  key  practices  are  the  tasks  within  a  core  element  that  must  be 
performed  by  an  organization  in  order  to  effectively  implement  and 
institutionalize  a  critical  process.  In  Section  5,  each  key  practice  is 
followed  by  commentary  about  the  key  practice  and  additional 
information  that  may  assist  the  organization  in  understanding  or 
interpreting  how  the  key  practice  could  be  implemented. 
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Figure  3.1 :  The  Components  of  an  ITIM  Critical  Process 


Principles  Guiding  the 
Use  and  Interpretation 
of  ITIM 


Regardless  of  the  specific  reason  for  using  ITIM,  the  following  principles8 
should  guide  each  interpretation  and  use  of  this  framework. 

ITIM  is  a  generic  framework  intended  for  broad  use.  Implementation  and 
improvement  needs  may  vary,  depending  on  the  specific  context. 


These  principles  were  derived  from  the  principles  found  in  SEI's  Software  Acquisition 
Capability  Maturity  Model. SH 
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•  HIM  is  a  framework  for  organizational  improvement.  Specifically,  HIM 
focuses  on  building  the  IT  investment  management  process  of  an 
organization. 

•  HIM  serves  as  an  improvement  roadmap  and  describes  the  characteristics 
of  an  IT  investment  management  process  that  one  would  expect  to  see  at 
each  maturity  stage.  The  maturity  stages  prescribe  the  order  of  processes 
to  improve,  but  not  how  an  organization  is  to  improve  its  processes. 

•  ITIM  describes  critical  processes  and  key  practices.  This  list  may  not  be 
exhaustive,  however.  Other  investment  management  process  components 
may  exist  and  could  be  considered  for  addition  to  this  framework  as 
greater  context  sensitivity  develops  to  the  issues  surrounding  the  process 
of  IT  investment  management. 

•  Critical  processes  are  typically  adopted  over  time.  Each  critical  process 
will  generally  go  through  a  step-by-step  evolution  of  introduction, 
adoption  and  development,  and  finally  full  implementation  within  an 
organization  as  the  organization  changes  and  modifies  necessary  functions 
and  operations  and  reaches  a  particular  maturity  stage  (see  figure  3.2). 


Figure  3.2:  Critical  Processes  Are  Typically  Introduced  at  a  Lower  Stage 
Before  Reaching  Full  Implementation 


Maturity  Stages 


Page  15 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Section  3 

Components  of  HIM 


•  HIM  does  not  address  all  the  factors  that  can  affect  investment  success. 
Examples  of  topics  excluded  from  HIM  are  strategic  planning,  funding 
availability,  and  specific  technology  implementations. 

•  HIM  takes  a  process  management  approach.  The  value  of  any  product  or 
service  is  largely  governed  by  the  quality  of  the  management  process  used 
to  create,  develop,  acquire,  and  maintain  it  and  by  the  direct  applicability 
of  the  product  or  service  to  achieving  the  organization's  strategic  plan. 

•  Any  process  can  be  improved]  continuous  improvement  efforts  are 
necessary  to  increase  efficiency  and  improve  effectiveness. 

•  There  is  no  "one  right  way" to  implement  HIM.  HIM  describes  the 
characteristics  of  mature  and  successful  IT  investment  management 
processes,  not  specific  implementation  techniques. 

•  ITIM  is  technology  independent.  For  example,  no  specific  tools,  methods, 
or  technologies  are  mandated  by  ITIM.  Appropriate  tools,  methods,  and 
technologies  should  be  made  available  to  support  the  processes  developed 
within  ITIM. 

•  Professional  judgment  must  be  applied  when  interpreting  ITIM  in  the 
context  of  a  particular  organization. 
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HIM  identifies  key  IT  investment  processes,  measures  the  presence  or 
absence  of  these  key  processes,  creates  an  assessment  of  an  organization's 
IT  investment  management  capability  and  maturity,  and  offers 
recommendations  for  improvement.  As  such,  ITIM  can  be  a  valuable  tool 
that  (1)  supports  organizational  self-assessment  and  improvement  and 
(2)  provides  a  standard  against  which  an  external  evaluation  of  an 
organization  can  be  conducted. 


ITIM  offers  organizations  a  roadmap  for  improving  their  IT  investment 
management  processes  in  a  systematic  and  organized  manner.  These 
process  improvements  are  intended  to 

•  improve  the  likelihood  that  IT  investments  will  be  completed  on  time  and 
on  budget, 

•  promote  a  better  understanding  and  management  of  IT-related  risks, 

•  ensure  that  IT  investments  are  selected  based  on  their  merits  by  a  well- 
informed  decision-making  body, 

•  implement  process  management  improvement  ideas  and  innovations,  and 

•  increase  the  business  value  and  mission  performance  improvements  of  IT 
investments. 

The  implementation  of  ITIM  as  a  tool  for  organizational  improvement  can 
be  achieved  in  a  variety  of  ways.  For  example,  an  organization  can  create  a 
separate  improvement  program,  employ  external  assistance  and  support, 
or  use  it  as  a  managerial  support  tool.  Regardless  of  the  implementation 
technique,  the  following  important  factors  should  be  considered  when 
using  ITIM  as  an  organizational  improvement  tool. 

•  Many  organizations  will  have  a  variety  of  selection,  control,  and  evaluation 
processes  currently  in  place  across  the  organization.  ITIM  can  help  these 
organizations  understand  the  relationships  among  these  processes  and 
determine  the  key  opportunities  for  immediate  improvements. 

•  ITIM  is  a  structured  approach  that  identifies  the  key  practices  for  creating 
and  maintaining  successful  IT  investment  management  processes. 
However,  ITIM  describes  what  to  do,  not  how  to  do  it.  Thus,  specific 
implementation  methods  can  and  will  vary  by  organization. 


ITIM  as  a  Tool  for 

Organizational 

Improvement 
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•  The  developmental  nature  of  a  maturity  model  means  that  process 
maturation  is  cumulative.  Lower  stage  processes  provide  the  foundation 
for  upper  stage  processes.  As  additional  critical  processes  are  introduced 
into  the  organization  and  implemented,  the  organization  attains  greater 
process  capabilities  and  maturity.  The  maturity  progression  also  means 
that  as  the  organization  incorporates  additional  processes  at  each 
successive  stage  of  maturity,  previously  implemented  lower  stage  critical 
processes  must  be  maintained. 

•  ITIM  is  not  a  substitute  for  good  project  management.  While  ITIM  takes  an 
enterprisewide  focus,  good  project-level  management  forms  the 
foundation  for  successful  IT  investments. 

•  Critical  processes  may  be  initially  implemented  and  practiced  within 
individual  bureaus  or  divisions  before  they  are  implemented  and  are 
mature  across  the  organization. 

•  Within  ITIM,  business  process  improvement  (BPI)  initiatives  are  not 
considered  to  be  IT  investments  but  instead  are  considered  to  be  parallel 
efforts  that  mayor  may  not  be  linked  to  IT  investments.  Thus,  ITIM 
assessments  do  not  evaluate  individual  BPI  initiatives.  However,  if  such 
initiatives  do  have  IT  investments,  then  these  IT  investments  should  be 
subject  to  the  organization's  IT  investment  management  process. 


ITIM  as  a  Tool  for 
Assessing  the  Maturity 
of  an  Organization 


J  ust  as  ITIM  can  be  used  as  a  tool  for  organizational  improvement,  it  can 
also  be  used  as  a  standard  against  which  the  IT  investment  management 
process  maturity  of  a  given  organization  can  bejudged.  For  example,  ITIM 
can  be  used  to  support  external  inspections  to  ensure  compliance  with 
industry  standards  or  acceptable  practices,  independent  reviews  of 
organizational  maturity  by  oversight  bodies,  or  other  external  IT  process 
reviews.  Regardless  of  the  specific  use,  however,  the  following  important 
factors  should  be  considered  when  using  ITIM  as  an  organizational 
assessment  tool. 


•  An  ITIM  assessment  can  be  conducted  for  an  entire  organization  (e.g.,  an 
executive  branch  department)  or  for  one  of  its  lower  level  divisions  (e.g.,  a 
branch,  bureau,  or  agency).  However,  the  unit  or  scope  of  analysis  (e.g., 
branch,  bureau,  agency,  or  department)  must  be  defined  before 
conducting  an  ITIM  assessment.  Additionally,  the  assessed  maturity  stage 
for  a  lower  level  division  is  not  necessarily  indicative  of  the  maturity  stage 
of  a  higher  level  division  or  of  the  organization  as  a  whole. 
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•  HIM  is  applicable  to  organizations  of  different  sizes.  Some  of  the 
processes  described  in  HIM  may  be  implicitly  conducted  by  smaller 
organizations.  For  example,  although  HIM  addresses  the  organizational 
need  to  align  and  coordinate  multiple  IT  investment  boards,  clearly  a 
smaller  organization  with  only  one  IT  investment  board  would  implicitly 
perform  this  critical  process. 

•  An  organization  may  be  concurrently  implementing  key  practices 
associated  with  several  maturity  stages.  In  fact,  key  practices  associated 
with  upper  stage  critical  processes  are  frequently  initiated  while  the 
organization  as  a  whole  is  at  a  lower  stage  of  maturity.  However, 
organizational  maturity  is  determined  by  assessing  at  what  maturity  stage 
the  organization  implements  all  key  practices  for  all  of  the  critical 
processes  associated  with  a  given  stage  of  maturity  and  any  lower 
maturity  stages.  For  example,  performing  key  practices  in  just  several 
Stage  3  critical  processes  does  not  mean  the  organization  has  attained 
Stage  3  maturity. 

•  The  key  practices  describe  what  is  to  be  done  not  how  it  is  to  be  done. 
Alternative  practices  may  accomplish  the  underlying  purpose  of  a  critical 
process.  The  key  practices  should  be  interpreted  rationally  to  judge 
whether  the  purpose  of  the  critical  process  is  effectively  achieved. 


Limitations  and 
Boundaries  of  ITIM 


ITIM,  like  other  assessment  tools,  has  its  limitations  and  boundaries.  For 
example,  while  strategic  planning  and  decisions  can  greatly  influence  the 
performance  of  an  organization,  ITIM  does  not  evaluate  strategic  plans 
and  decisions  made  by  the  organization's  executives.  Rather  the  purpose 
of  ITIM  is  to  describe  and  improve  the  IT  investment  management 
processes  so  that  the  strategic  plans  and  decisions  that  are  made  can  and 
will  be  effectively  supported  by  highly  effective  IT  investments. 

Similarly,  performance  measures  created  and  used  to  guide  the 
organization  and  its  activities  are  a  factor  in  some  ITIM  processes  and  can 
be  viewed  as  maturing  in  parallel  to  the  IT  investment  management 
processes.  However,  in  general,  activities  related  to  the  ongoing 
development  and  implementation  of  performance  measures  are  largely 
outside  the  scope  of  ITI M  ,9 


9For  additional  guidance  on  developing  performance  measures,  see  Executive  Guide:  Measuring 
Performance  and  Demonstrating  Results  of  Information  Technology  Investments  (GAO/AIM  D-98-89, 
March  1998). 
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Also,  HIM  does  not  address  IT  acquisition  (e.g.,  which  type  of  contract  to 
use  or  how  best  to  conduct  price  negotiations,  etc.)  as  a  separate 
investment  management  step.  While  important,  the  primary  purpose  of 
acquisition-related  activities  is  to  support  the  execution  of  the  IT 
investment  decisions  that  are  made  by  the  IT  investment  board(s).10Thus, 
one  would  expect  that  the  acquisition  aspects  of  project  development 
would  be  embedded  in  the  IT  project  proposal  and  analysis  steps  within 
ITIM.  Additionally,  the  acquisition  strategy  might  be  part  of  the  project's 
risk  assessment  (i.e.,  the  risks  of  pursuing  various  acquisition 
alternatives). 

Finally,  individuals  selecting  ITIM  as  an  assessment  tool  should  do  the 
following: 

•  Become  proficient  with  the  related  GAO  and  OMB  IT  investment  guidance 
mentioned  in  the  introductory  section.  This  is  particularly  true  for  those 
seeking  to  apply  ITIM  in  the  federal  government  sector.  Understanding 
this  guidance  provides  greater  insight  into  the  developmental  history,  key 
issues,  and  critical  success  factors  associated  with  the  IT  investment 
approach. 

•  Become  familiar  with  generally  accepted  capital  decision-making 
approaches  and  associated  analytical  tools. 

•  Receive  maturity  model  training  to  become  familiar  with  the  basic 
concepts  behind  maturity  models. 

•  Have  experience  assessing  organizations  using  standardized  assessment 
tools. 


10  For  more  information  on  procurement  within  the  context  of  a  capital  budget,  see  OMB's 
Capital  Programming  Guide,  Version  1.0  (J  uly  1997). 
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Figure  5.1  shows  the  five  ITIM  stages  of  maturity  and  the  critical  processes 
that  define  each  maturity  stage. 


Figure  5.1 :  The  ITIM  Stages  of  Maturity  With  Critical  Processes 


Maturity  Stages 


Critical  Processes 


a.  Investment  Process  Benchmarking 
□  IT-Driven  Strategic  Business  Change 


□  Post-Implementation  Reviews  and  Feedback 

□  Portfolio  Performance  Evaluation  and  Improvement 

□  Systems  and  Technology  Succession  Management 

Authority  Alignment  of  IT  Investment  Boards 

□  Portfolio  Selection  Criteria  Definition 
Investment  Analysis 

5jS  Portfolio  Development 
^’Portfolio  Performance  Oversight 


IT  Investment  Board  Operation 

□  IT  Project  Oversight 
IT  Asset  Tracking 

Business  Needs  Identification  for  IT  Projects 

□  Proposal  Selection 

IT  Spending  without  Disciplined  Investment 
Processes 


The  following  subsections  describe  each  ITIM  maturity  stage  in  greater 
detail.  The  first  subsection  only  describes  the  attributes  of  ITIM  Stage  1, 
since  no  critical  processes  are  associated  with  this  stage.  Each  following 
subsection  describes  one  of  the  ITIM  stages.  In  each  subsection,  the  ITIM 
stage  is  briefly  introduced  and  its  associated  critical  processes  are 
identified  along  with  a  list  of  applicable  criteria.  For  each  critical  process, 
a  brief  introduction  is  presented  along  with  a  map  depicting  the  associated 
core  elements  (purpose,  organizational  commitment,  prerequisites, 
activities,  and  evidence  of  performance)  and  key  practices  for  the  critical 
process.  Following  the  map,  each  core  element  presents  the  associated 
key  practices  (printed  in  bold  text)  and  a  discussion  and  interpretation  of 
the  key  practice.  For  ease  of  use  as  a  reference  document,  the  page 
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headings  for  section  5  indicate  which  stage  and  critical  processes  is  being 
discussed  on  each  page. 


ITIM  Stage  1:  Creating 
I  nvestment  Awareness 


The  following  section  provides  a  description  of  the  conditions  and 
characteristics  associated  with  an  organization  operating  at  ITIM  Stage  1. 
Within  ITIM,  Stage  1  is  different  from  the  other  maturity  stages  in  that: 


•  it  is  assumed  to  be  the  default  stage  for  an  organization  that  has  not 
undergone  an  ITIM  assessment, 


•  there  are  no  critical  processes  associated  with  Stage  1,  and 


•  it  is  typified  by  the  absence  of  an  organized,  executable,  and  consistently 
applied  IT  investment  management  process. 

The  following  description  of  an  ITIM  Stage  1  organization  is  not  intended 
to  be  comprehensive;  rather,  it  provides  an  overview  of  the  general 
conditions  and  problems  that  typically  confront  a  Stage  1 
organization.Overall,  an  ITIM  Stage  1  organization  has  ad  hoc  or 
undisciplined  IT  investment  management  processes.  This  often 
contributes  to  escalating  project  costs,  unmitigated  risks,  frequent  project 
schedule  slippages,  and  low  value  mission  or  business  benefits. 

F  urthermore,  while  the  organization  may  have  "pockets  of  excellence"  in 
IT  investment  management,  the  variability  in  these  processes  across  the 
organization  results  in  inconsistency  in  IT  project  outcomes  and  results. 


Selection  Process  The  Stage  1  organization's  focus  is  more  often  on  a  project's  funding 

requirements  and  lower  level  organizational  requirements  rather  than  on 
(1)  its  value  toward  achieving  agency  mission  goals,  (2)  its  technical  and 
economic  risks,  (3)  its  performance  problems,  or  (4)  cost  and  schedule 
overruns.  IT  is  treated  largely  as  an  expense  item  in  the  budget  and  may  be 
intertwined  with  other  administrative  and  management  support  funding 
needs.  Also,  multiyear  IT  projects  that  are  "in  the  budget  pipeline"  are 
reviewed  each  year  largely  in  terms  of  marginal  increases  or  decreases  to 
the  previous  year's  funding  base,  regardless  of  cost,  schedule,  and 
performance  results  to  date. 

In  short,  while  some  IT  projects  within  a  Stage  1  organization  may  be 
funded  because  they  link  to  a  defined  business  or  mission  purpose,  many 
projects  are  funded  despite  the  absence  of  critical  information  that 
demonstrates  expected  and  achieved  improvements  in  program,  business, 
or  mission  performance. 
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Control  Process  Stage  1  organizations  typically  have  unstructured,  ill-timed,  and 

inconsistent  IT  investment  management  controls.  Senior  executives  and 
line  managers  may  rarely  review  IT  projects'  performance  data  and  thus, 
the  organization  lacks  an  early  warning  method  to  quickly  detect  and 
rectify  major  problems.  Instead,  project  crises  are  handled  as  they  arise, 
focusing  only  on  quick  fixes  rather  than  considering  any  systemic  causes 
of  the  problems.  As  a  result,  individual  project  success  is  unpredictable 
and  may  largely  be  the  result  of  extraordinary  individual  or  project  team 
efforts. 

Additionally,  a  Stage  1  organization  rarely  would  have  an  up-to-date  and 
complete  inventory  of  its  IT  assets.  For  example,  although  it  may  have  an 
IT  hardware  (equipment)  inventory,  the  organization  might  lack  a 
comprehensive  list  of  systems,  software  applications  and  tools,  or 
licensing  agreements.  Such  an  incomplete  asset  inventory  precludes  an 
adequate  investment  control  process. 


Evaluation  Process  Finally,  a  Stage  lorganization  rarely,  if  ever,  (1)  evaluates  IT  investment 

outcomes  or  (2)  identifies  lessons  learned  from  the  projects.  If  such 
evaluations  are  conducted,  they  tend  to  be  poorly  staffed,  conducted 
without  a  formal  process  that  delineates  method,  scope,  and 
responsibilities,  and  often  are  triggered  only  in  response  to  outside 
pressures  (e.g.,  an  audit  or  a  budget  oversight  review). 
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ITIM  Stage  2:  Building 
the  Investment 
Foundation 


Maturity  Stages 


Critical  Processes 


□  Investment  Process  Benchmarking 
IT-Driven  Strategic  Business  Change 


ra  Post-Implementation  Reviews  and  Feedback 
^Portfolio  Performance  Evaluation  and  Improvement 
Systems  and  Technology  Succession  Management 

Authority  Alignment  of  IT  Investment  Boards 
^Portfolio  Selection  Criteria  Definition 

□  Investment  Analysis 

□  Portfolio  Development 

□  Portfolio  Performance  Oversight 


IT  Investment  Board  Operation 
IT  Project  Oversight 
IT  Asset  Tracking 

Business  Needs  Identification  for  IT  Projects 
□  Proposal  Selection 

IT  Spending  without  Disciplined  Investment 
Processes 


Stage  2  builds  the  foundation  for  current  and  future  IT  investment  success 
by  establishing  mature  IT  control  processes  and  basic  IT  selection 
processes.  As  such,  this  stage  is  defined  by  the  following  five  critical 
processes: 

•  IT  Investment  Board  Operation  is  the  process  for  creating  and  defining 
one  or  more  IT  investment  boards  within  the  organization. 

Criteria:  Assessing  Risks  and  Returns:  A  Guide  for  Evaluating  Federal 
Agencies’  IT  Investment  Decision-making  (hereafter  referred  to  as  IT 
Assessment  Guide)  (AIMD-10.1.13),  p.  32,  (CCA,  OMB  M -97-0(2)); 
Executive  Guide:  Improving  Mission  Performance  Through  Strategic 
Information  Management  and  Technology!  hereafter  referred  to  as  SIM 
Executive  Guide)  (AIM  D-94-115),  Practices  2, 10;  Evaluating  Information 
Technology  Investments,  version  1.0,  (hereafter  referred  to  as  OMB  IT 
Investment  Guide)  Office  of  Management  and  Budget,  p.  3;  Capital 
Programming  Guide,  version  1.0,  Office  of  Management  and  Budget,  p.  ii. 

•  IT  Project  Oversight  is  a  pivotal  process  whereby  the  organization 
monitors  all  projects  relative  to  costand  schedule  expectations. 
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Criteria:  IT  Assessment  Guide  (AIM  D-10.1.13),  p.  52,  (CCA,  PRA,  FASA, 

EO  13011,  OMB  A-ll,  Part  3);  OMB  IT  Investment  Guide,  p.  10. 

•  IT  Asset  T racking  is  the  process  by  which  the  IT  inventory  is  created 
and  maintained  to  provide  asset  tracking  data  to  executive 
decisionmakers. 

Criteria:  IT  Assessment  Guide  (AIM  D-10.1.13),  p.  8, 19;  PRA;  E.0. 13103; 
Capital  Programming  Guide,  p.  ii. 

•  Business  Needs  Identification  for  IT  Projects  is  the  process  for 
identifying  the  key  customers  (or  end  users)  and  near-term  business  needs 
that  each  IT  project  will  support. 

Criteria:  IT  Assessment  Guide  (AIM  D-10.1.13),  p.  15, 16, 17;  S/M  Executive 
Guide  [AIM  D-94-115],  Practices  4,  9;  OMB  M-97-16. 

•  Proposal  Selection  -  introduces  an  organization  to  defined  processes 
used  to  select  new  IT  project  proposals. 

Criteria:  Based  on  IT  Assessment  Guide  (AIM  D-10.1.13),  p.  23-25,  (CCA, 
PRA,  EO  13011,  OMB  A-ll,  OMB  A-130,  OM B  A-109,  OMB  A-94, 

OMB  M -97-0(2)) 
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IT  Investment  Board 
Operation 


The  IT  investment  board  is  a  key  component  in  the  IT  investment 
management  process.  This  critical  process  defines  the  membership, 
guiding  policies,  operations,  roles,  responsibilities,  and  authorities  for 
each  designated  board  and,  if  appropriate,  each  board's  support  staff.  This 
definition  provides  the  basis  for  each  board's  IT  investment  selection, 
control,  and  evaluation  activities  throughout  this  maturity  model. 

Depending  on  its  size,  structure,  and  culture,  an  organization  may  have 
more  than  one  IT  investment  board.  This  critical  process  is  based  on  the 
assumption,  that  for  managerial  reasons,  the  key  practices  in  this  critical 
process  will  be  implemented  consistently  across  each  of  these  boards  and 
that  the  organization  will  tailor  the  board's  operations  as  part  of 
implementing  this  critical  process. 
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Figure  5.2:  IT  Investment  Board  Operation 
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Purpose 


Organizational  Commitment 


To  define  and  establish  the  governing  board(s)  responsible  for 
selecting,  controlling,  and  evaluating  IT  investments. 


Commitment  1:  An  organization-specific  IT  investment  process 
guide  is  created  to  direct  each  board's  operations. 

Each  organization  must  take  the  available  general  IT  investment  process 
guidance11  and  define  the  unique  manner  in  which  this  guidance  will  be 
implemented  within  the  organization.  This  process  guide  should  include 

•  specifics  about  the  roles  of  key  people  within  its  IT  investment  processes; 

•  an  outline  of  the  significant  events  and  decision  points  within  the 
processes; 

•  an  identification  of  the  external  and  environmental  factors  that  will 
influence  the  processes  (i.e.,  legal  constraints,  the  behavior  of  key 
suppliers  or  customers,  or  industry  norms);  and 

•  the  manner  in  which  IT  investment-related  processes  will  be  coordinated 
with  other  organizational  plans  and  processes. 

This  process  guidewill  be  a  key  document  that  the  organization  will  use  to 
initiate  and  manage  its  IT  investment  processes.  For  example,  this  guide 
forms  the  foundation  for  each  IT  board's  operating  policies  and 
procedures  and  can  also  serve  as  the  foundation  for  many  of  the  policies 
that  are  required  in  many  other  critical  processes  within  ITIM. 

This  process  guide  can  serve  multiple  purposes.  For  example,  it  can  serve, 
in  part  or  in  whole,  as  the  document  for  which  the  other  required  policies 
in  ITIM  are  based  (e.g.,  the  policy  for  setting  up  and  managing  the  IT  asset 
inventory).  Additionally,  this  process  guide  can  be  the  basis  for  any  other 
IT  management  policies  and  procedures  beyond  the  key  ones  identified  in 
ITIM.  An  example  of  other  management  procedures  would  be  the 
development  of  an  initial  IT  project  screening  mechanism  used  by  larger 


nAssessing  Risks  and  Returns:  A  Guide  for  Evaluating  Federal  Agencies'  IT  Investment  Decision- 
mak/r)g(GAO/AIMD-10.1.13,  February  1997);  Executive  Guide:  Improving  Mission  Performance  Through 
Strategic  Information  Management  and  Technology  (GAO/AIMD-94-115,  May  1994);  Evaluating 
Information  Technology  Investments,  A  Practical  Guide,  Executive  Office  of  the  President,  Office  of 
Management  and  Budget,  November  1995.  Capital  Programming  Guide,  version  1.0,  Office  of 
Management  and  Budget,  (J  uly  1997). 
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organizations  to  ensure  that  each  IT  project  is  sufficiently  complete  before 
being  reviewed  by  the  IT  investment  board. 

Commitment  2:  Organization  executives  and  line  managers  support 
and  carry  out  IT  investment  board  decisions. 

For  each  IT  investment  board  to  be  effective,  it  must  have  the  formal, 
acknowledged  support  of  the  organization's  executives  and  line  managers 
and  these  managers  must  execute  the  board's  decisions.  Examples  of  this 
organizational  support  may  be  indicated  by 

•  language  in  executive  employment  contracts; 

•  memoranda  between  executives  and  subordinate  line  managers;  and 

•  formal,  signed  policy  endorsement  by  executives  and  managers. 


Prerequisites  Prerequisite  1:  Adequate  resources  are  provided  for  operating  each 

IT  investment  board. 

These  resources  typically  involve 

•  top  management  participation  in  creating  the  board(s)  and  defining  their 
scope, 

•  resources  and  staff  support  (including  external  experts  or  process 
advisors)  to  support  the  execution  of  this  critical  process,  and 

•  an  investment  management  center  that  can  benefit  both  the  IT  investment 
board  and  IT  project  managers. 

Prerequisite  2:  Board  members  understand  the  investment  board's 
policies  and  procedures  and  exhibit  core  competencies  in  using  the 
IT  investment  approach  via  training,  education,  or  experience. 

Board  members  should  understand  the  board's  policies,  roles,  rules,  and 
activities  and  be  capable  of  carrying  out  their  responsibilities  competently. 
Thus,  education  and  training  for  members  with  little  or  no  investment 
decision-making  experience  is  needed  in  areas  such  as  economic 
evaluation  techniques,  capital  budgeting  methods,  performance 
measurement  strategies,  and  risk  management  approaches. 

Knowledge  building  and/or  training  may  include 


Page  31 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Section  5:  Critical  Processes  For  The  ITIM  Stages 
□  ITIM  Stage  2:  Building  the  Investment  Foundation 
□□□  IT  Investment  Board  Operation 


•  courses  specifically  designed  for  new  members, 

•  educational  forums, 

•  formal  seminars,  or 

•  executive  training  programs  offering  in-depth  courses. 


Activities  Activity  1:  Each  IT  investment  board  is  created  and  defined  with 

board  membership  integrating  both  IT  and  business  knowledge. 

The  organization  creates  and  documents  the  prescribed  activities  of  the  IT 
investment  board(s).  The  investment  board(s)  should 

•  have  final  project  funding  decision  authority  over  (or  provide  a  direct 
recommendation  to  the  agency  head  for)  projects  within  their  scope  of 
authority, 

•  be  comprised  of  key  business  unit  executives  and  business  support 
executives  (i.e.,  financial  management  and  information  systems 
executives),  and 

•  ensure  executive  sponsorship  and  responsibility  for  the  organization's 
major  IT  projects  and  investments. 

An  organization  may  also  create  IT  investment  boards  at  other 
organizational  tiers  that,  for  example,  correspond  to  its  business  or 
mission  area  structure.  The  policies  and  procedures  that  describe  the  roles 
of  these  boards  may  be  addressed  as  a  precursor  to  the  Stage  3  critical 
process  "Authority  Alignment  of  IT  Investment  Boards." 

Additionally,  each  defined  board  (particularly  in  a  larger  organization) 
may  wish  to  create  one  or  more  working  groups  to  carry  out  the 
authorized  activities  of  the  board.  However,  the  boards  themselves  are 
ultimately  responsible  for  the  execution  of  their  designated  activities. 

Activity  2:  Each  IT  investment  board  operates  according  to  written 
policies  and  procedures  in  the  organization-specific  IT  investment 
process  guide. 

The  board's  work  processes  and  decision-making  processes  (i.e., 
schedules,  agendas,  authorities,  decision-making  rules,  etc.)  are  described 
and  documented.  The  board  should  be  an  active  decision-making  body 
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meeting  regularly  (e.g.,  monthly  or  quarterly).  Project  funding  decision¬ 
making  should  occur  at  least  once  a  year.  The  mechanics  of  the  decision¬ 
making  processes  should  be  as  simple  and  comprehensible  as  possible 
while  taking  into  account  the  activities  needed  for  the  board  to  be 
effective. 

Examples  of  output  from  the  IT  investment  board  may  include 

•  project  funding  decision  documents, 

•  executive  actions  memorandums, 

•  project  review  decisions,  and 

•  board  meeting  minutes. 

Evidence  1:  Physical  evidence  of  IT  Investment  Board  Operation 
exists. 

Physical  evidence  could  include,  for  example: 

•  board  meetings, 

•  working  group  meetings,  and 

•  board  member  training  classes. 

Evidence  2:  Documentary  evidence  of  IT  Investment  Board 
Operation  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example: 

•  standard  policies  and  procedures, 

•  an  organization-specific  IT  investment  process  guide, 

•  board  meeting  minutes  including  attendance,  discussions,  and  decisions, 

•  project  review  decision  papers, 

•  decisional  documents  and  memorandums, 

•  executive  action  memoranda  between  the  board  and  subordinate  line 
managers,  and 
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•  a  formal,  signed  policy  endorsement  by  executives  and  managers. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  IT  Investment  Board  Operation. 

Testimonial  evidence  could  include,  for  example: 

•  board  member  interviews  and 

•  working  group  member  interviews. 
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The  purpose  of  this  critical  process  is  to  ensure  that  the  organization 
provides  effective  oversight  for  its  IT  projects  throughout  all  phases  of 
their  life  cycle.  While  the  board  should  not  micromanage  each  project  in 
order  to  provide  effective  oversight,  it  should  maintain  adequate  visibility 
over  performance  and  progress  and  use  this  visibility  to  review  each 
project's  progress  toward  predefined  cost  and  schedule  expectations  as 
well  as  anticipated  benefits  and  risk  exposure.  The  board  should  expect 
that  each  project  development  team  is  responsible  for  meeting  project 
milestones  within  the  expected  cost  parameters  established  by  the 
project's  business  case  and  cost/benefit  analysis.  The  board  should  also 
employ  early  warning  systems  that  enable  it  to  take  corrective  actions  at 
the  first  sign  of  cost,  schedule,  and  performance  slippages. 

The  cognizant  IT  investment  board  has  ultimate  responsibility  for  the 
activities  within  this  critical  process.  However,  in  larger  organizations  the 
board  may  authorize  designated  subgroups  to  carry  out  some  of  these 
activities. 


Page  36 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Section  5:  Critical  Processes  For  The  ITIM  Stages 
□  ITIM  Stage  2:  Building  the  Investment  Foundation 
□□□  IT  Project  Oversight 


Figure  5.3:  IT  Project  Oversight 


Purpose 

To  regularly  determine  each  IT  project’s  progress  toward 
cost  and  schedule  milestones  using  established  criteria  and 
take  corrective  actions  when  milestones  are  not  achieved. 


Prerequisites 

1 .  Adequate  resources  are 
provided  to  assist  the  board(s)  in 
overseeing  IT  projects. 

2.  Each  IT  project  has  and 
maintains  an  approved  project 
management  plan  that  includes 
cost  and  schedule  controls. 

3.  An  IT  investment  board  is 
operating. 

4.  Information  from  the  IT  asset 
inventory  is  used  by  the  IT 
investment  board  as  applicable. 


Activities 

1 .  Each  project's  up-to-date  cost  and 
schedule  data  are  provided  to  the 
appropriate  IT  investment  board. 

2.  Using  established  criteria,  the  IT 
investment  board  oversees  each  IT 
project’s  performance  regularly  by 
comparing  actual  cost  and  schedule  data 
to  expectations. 

3.  The  IT  investment  board  performs 
special  reviews  of  projects  that  have  not 
met  predetermined  performance 
standards. 

4.  Appropriate  corrective  actions  for  each 
underperforming  project  are  defined, 
documented,  and  agreed  to  by  the  IT 
investment  board  and  the  project 
manager. 

5.  Corrective  actions  are  implemented 
and  tracked  until  the  desired  outcome  is 
achieved. 


Evidence  of 
Performance 

1 .  Physical  evidence  of  IT 
Project  Oversight  exists. 

2.  Documentary  evidence  of 
IT  Project  Oversight  is  created 
and  maintained. 

3.  Testimonial  evidence  is 
made  available  during  reviews 
of  IT  Project  Oversight. 


Organizational  Commitment 

1 .  The  organization  has  written  policies  and  procedures  for 
project  management. 

2.  The  organization  has  written  policies  and  procedures  for 
management  oversight  of  IT  projects. 
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To  regularly  determine  each  IT  project's  progress  toward  cost  and 
schedule  milestones  using  established  criteria  and  take  corrective 
actions  when  milestones  are  not  achieved. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  project  management. 

These  policies  and  procedures  typically  specify  the  following: 

•  A  documented  project  management  plan  for  the  entire  life  cycle  is 
developed,  used,  and  maintained  as  the  basis  for  tracking  the  project.  The 
project  management  plan  incorporates  other  plans  such  as  a  software 
development  plan  or  system  integration  and  test  plan. 

•  The  project  manager  is  continually  aware  of  the  project's  status  and 
associated  development  and  acquisition  issues. 

Commitment  2:  The  organization  has  written  policies  and 
procedures  for  management  oversight  of  IT  projects. 

These  policies  and  procedures  typically  specify  the  following: 

•  Each  IT  investment  board's  responsibilities  when  performing  project 
oversight  activities  within  its  domain. 

•  The  procedural  rules  for  IT  investment  board  operation  and  decision¬ 
making  during  project  oversight. 

•  The  threshold  criteria  that  the  IT  investment  board(s)  uses  when  analyzing 
actual-versus-expected  project  performance  as  part  of  its  oversight 
function.  This  threshold  is  typically  defined  on  the  basis  of  the  cost  or 
schedule  measures  (e.g.,  currently  more  than  10  percent  over  expected 
cost).  This  predefined  threshold  will  be  a  major  factor  in  determining 
remedial  actions. 

•  Decisions  that  are  required  when  the  project  deviates  or  varies 
significantly  from  the  project  management  plan. 

•  Changes  to  the  project's  commitments  are  made  with  the  involvement  of 
affected  groups.  Examples  of  these  affected  groups  include 

•  system  engineering, 
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•  software  engineering  (including  all  subgroups,  such  as  software 

design), 

•  hardware  engineering, 

•  project  planning  and  estimating, 

•  project  stakeholders  and  champions, 

•  business  units,  and 

•  customers  and  end  users. 

•  Each  IT  investment  board  oversees  all  project  commitment  changes  and 
new  project  commitments  made  to  individuals  and  groups  external  to  the 
organization. 

•  The  responsibilities  of  the  project  manager. 

Prerequisites  Prerequisite  1:  Adequate  resources  are  provided  to  assist  the 

board(s)  in  overseeing  IT  projects. 

These  resources  can  include 

•  a  manager  and  staff  to  be  assigned  specific  responsibilities  for  monitoring 
IT  projects  and 

•  tools  to  support  board(s)  operations  may  include  project  metric  summary 
reports  and  decision  support  applications. 

Prerequisite  2:  Each  IT  project  has  and  maintains  an  approved 
project  management  plan  that  includes  cost  and  schedule  controls. 

Each  IT  project  management  team  creates  and  maintains  a  project 
management  plan.12  This  plan  documents  a  variety  of  project  decisions, 
assumptions,  and  expectations  including  the  project  performance 
expectations.13  Part  of  these  expectations  could  include  a  cost  and 
schedule  baseline  control  system  such  as  earned  value  management 
system,  milestone-based  accomplishment  expectations,  or  other  such 


12See  IEEE  1058  Standard  for  Software  Project  Management  Plans  for  an  example  of 
additional  guidance  on  creating  a  project  management  plan. 

13See  Executive  Guide:  Measuring  Performance  and  Demonstrating  Results  of  Information 
Technology  Investments  (GAO/AIMD-98-89,  March  1998)  for  additional  guidance  on 
performance  measurement. 
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control  system  as  is  commensurate  with  the  project's  size,  importance, 
cost,  and  risk.14 

Prerequisite  3:  An  IT  investment  board  is  operating. 

An  investment  board  has  the  primary  management  oversight  responsibility 
and  is  the  central  decision-making  body  in  this  critical  process. 

(See  also  Stage  2-IT  I  nvestment  Board  Operation  for  a  description  of  the 
roles  and  responsibilities  of  the  investment  board.) 

Prerequisite  4:  Information  from  the  IT  asset  inventory  is  used  by 
the  IT  investment  board  as  applicable. 

The  asset  inventory  is  necessary  to  ensure  that  each  board  is  aware  of  all 
of  the  IT  projects  and  resources  for  which  it  is  responsible. 

(See  also  Stage  2-IT  Asset  T racking  for  a  description  of  the  activities 
associated  with  developing  an  IT  asset  inventory). 

Activities  Activity  1:  Each  project's  up-to-date  cost  and  schedule  data  are 

provided  to  the  appropriate  IT  investment  board. 

The  cost  and  schedule  data  (both  expected  and  actual)  for  each  IT  project 
are  collected  and  distributed  to  the  appropriate  IT  investment  board. 
These  data  may  be  collected  by  the  board  itself  or  collected  and 
distributed  in  some  other  manner  (i.e.,  through  a  centralized  third  party). 
These  data  will  be  key  to  assisting  each  cognizant  IT  board  in  its  decision¬ 
making. 

Activity  2:  Using  established  criteria,  the  IT  investment  board 
oversees  each  IT  project's  performance  regularly  by  comparing 
actual  cost  and  schedule  data  to  expectations. 

The  board  typically  oversees  the  project's  performance  periodically  or  at 
major  milestones  to  interpret  the  project  cost  and  schedule  status  data 
with  respect  to  historic  project  data  and  project  expectations. 


“See  the  Defense  Department's  Earned  Value  Management  Web  site  at 
http://www.acq.osd.mil/pm  and  Planning,  Budgeting,  and  Acquisition  of  Capital  Assets, 
(Office  of  Management  and  Budget  Circular  No.  A-ll,  Part  3,  J  uly  1999)  for  additional 
guidance  on  earned  value  management. 
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Project  oversight 

•  is  conducted  at  least  at  the  major  life-cycle  milestones  for  all  projects; 

•  differs  in  its  degree  of  depth  depending  on  the  size,  cost,  and  importance 
of  the  project; 

•  must  compare  estimated  schedule  time  frames  versus  actual  schedules, 
including  schedule  slippage  and/or  compressions; 

•  must  compare  estimated  costs  versus  costs  spent  or  obligated  to  date,  any 
changes  in  funding,  and  the  impact  of  these  changes,  and 

•  may  have  other  staff  attending  such  as  an  independent  audit  team,  quality 
assurance  group,  or  an  IV&V  contractor  who  is  responsible  for  ensuring 
that  project  information  is  valid  and  verifying  that  corrective  actions  have 
been  taken. 

Project  oversight  should  also  address  each  of  these  project  management 
issues: 

•  Method-problems  that  have  arisen  concerning  the  project  development 
methodology  (including  contractor  management  issues). 

•  Technical-technical  issues  or  problems  concerning  such  components  as 
hardware,  software,  or  telecommunications. 

•  Business/proiect  alignment-evaluation  of  benefits  delivered  to  date  and 
relationship  of  the  project  to  specific  business  objectives. 

•  Rjsks- assessment  of  the  risks  encountered  to  date  and  how  expected  risks 
are  to  be  managed. 

Activity  3:  The  IT  investment  board  performs  special  reviews  of 
projects  that  have  not  met  predetermined  performance  standards. 

Using  estimated  and  actual  cost  and  schedule  data,  the  organization 
should  identify  projects  that  are  not  meeting  their  cost  and/or  schedule 
performance  expectations.  The  following  are  examples  of  data  that  could 
be  compared: 

•  actual  cost  data  to  planned  cost  data; 

•  results  for  the  current  lifecycle  phase  to  expected  life  cycle  performance; 
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•  the  current  number  and  scope  of  requirements  to  the  original 
requirements  established  for  the  project; 

•  the  current  conditions  and  assumptions  to  the  projects'  initial  assumptions 
and  context;  and 

•  the  actual  performance  of  the  software  development  organization  to  their 
specified  deliverables  (e.g.,  schedule,  costs,  functionality,  technical 
solutions). 

Executives  should  ensure  that  there  are  incentives  for  identifying  and 
raising  problems  to  the  appropriate  decision-making  level  and  that  there 
are  no  incentives  for  covering  up  significant  problems. 

Activity  4:  Appropriate  corrective  actions  for  each 
underperforming  project  are  defined,  documented,  and  agreed  to 
by  the  IT  investment  board  and  the  project  manager. 

The  IT  investment  board  should  decide  on  corrective  actions  to  apply  to 
each  project  for  which  deficiencies  or  problems  have  been  identified  (e.g., 
actual  costs  exceed  estimated  costs,  the  schedule  has  slipped, 
requirements  have  changed). 

Corrective  actions  will  usually  involve  one  of  the  following  alternatives: 

•  modifying  the  project  (e.g.,  objectives,  scope,  deliverables,  time  frames); 

•  working  only  on  one  module  and  stopping  work  on  the  rest  of  the  project 
(until  a  milestone  is  reached  or  as  not  to  exceed  a  set  period  of  time); 

•  temporarily  stopping  the  project  to  permit  external  work  (e.g.,  related 
projects)  to  be  completed  (until  a  milestone  is  reached  or  not  to  exceed  a 
set  period  of  time); 

•  canceling  the  project;  or 

•  accelerating  the  project's  development  (e.g.,  accelerating  hardware 
deployment  across  the  organization). 

F  uture  "cascading"  actions  resulting  from  these  decisions  should  be 
clearly  identified  for  decisionmakers  to  consider  when  choosing  a 
corrective  action. 
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Evidence  of  Performance 


Activity  5:  Corrective  actions  are  implemented  and  tracked  until 
the  desired  outcome  is  achieved. 

The  investment  board  ensures  that 

corrective  actions  and  related  efforts  are  executed  by  the  project 
management  team  and  tracked  by  the  investment  board  until  the  desired 
outcomes  occur  and 

if  the  corrective  actions  are  significant  enough,  an  independent  review 
may  be  conducted  prior  to  returning  to  the  original  project  plan  (i.e., 
reinstatement  of  funding)  to  ensure  that  all  corrective  actions  have 
achieved  the  intended  results  and  to  determine  whether  additional 
changes  or  modifications  are  still  needed. 

Indicators  of  this  activity  might  include 

a  record  of  underperforming  projects  being  satisfactorily  corrected, 
evidence  of  unrecoverable  projects  being  terminated, 
a  record  of  independent  review  results  and  follow-up  action  plans,  or 
accelerated  roll-out  of  implementation. 

Evidence  1:  Physical  evidence  of  IT  Project  Oversight  exists. 

Physical  evidence  could  include,  for  example 

the  collection  and  delivery  of  each  project's  cost  and  schedule  data  to  the 
board, 

a  board  review  of  a  project's  cost  and  schedule  data  (e.g.,  meeting  minutes 
or  review  summaries),  and 

IT  investment  board  decisions  on  corrective  actions. 

Evidence  2:  Documentary  evidence  of  IT  Project  Oversight  is 
created  and  maintained. 

Documentary  evidence  could  include,  for  example 

a  written  project  management  policy, 

a  written  policy  for  management  reviews  of  IT  projects, 
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•  approved  project  management  plans, 

•  historic  project  data  and  project  expectations, 

•  corrective  action  memoranda,  and 

•  IT  asset  inventory  reports. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  IT  Project  Oversight. 

Testimonial  evidence  could  include,  for  example 

•  interviews  of  other  staff  (such  as  an  independent  audit  team,  quality 
assurance  group,  or  IV&V  contractor)  in  attendance  at  project  reviews  and 

•  project  manager  interviews. 
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IT  Asset  Tracking 


To  make  good  IT  investment  decisions,  an  organization  must  know  where 
its  IT  assets  (i.e.,  personnel,  systems,  applications,  hardware,  software 
licenses,  etc.)  are  located  and  how  funds  are  being  expended  toward 
acquiring,  maintaining,  and  deploying  these  assets.  This  critical  process 
identifies  IT  assets  within  the  organization  and  creates  a  comprehensive 
inventory  of  them.  This  inventory  is  used  to  track  the  organization's  IT 
resources  to  provide  insights  and  trends  about  major  IT  cost  and 
management  drivers. 

This  inventory  can  take  many  forms  (e.g.,  a  catalog,  list,  or  a  balance 
sheet),  but  regardless  of  form,  the  inventory  should  identify  each  IT  asset 
and  its  associated  components.  This  inventory  does  not  have  to  be 
centrally  located;  it  can  be  managed  on  a  distributed  basis.  The  guiding 
principle  for  developing  the  inventory  is  that  it  should  be  accessible  where 
it  is  of  the  most  value  to  IT  investment  decisionmakers.  The  inventory  is 
particularly  important  when  executing  the  IT  Project  Oversight,  Proposal 
Selection,  Investment  Analysis,  and  Systems  and  Technology  Succession 
Management  critical  processes.  Additionally,  beyond  serving  as  a  tool  to 
aid  in  IT  investment  decision-making,  the  IT  asset  inventory  can  also  assist 
the  organization  with  software  licensing  management,  hardware  life  cycle 
management,  and  system  architecture  plans. 
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Figure  5.4:  IT  Asset  Tracking 
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To  create  and  maintain  an  IT  asset  inventory  to  assist  in 
managerial  decision-making. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  developing  and  maintaining  an  IT  asset  inventory. 

These  policies  and  procedures  typically  specify: 

•  that  responsibility  for  submitting,  updating,  and  maintaining  relevant 
inventory  information  for  each  project  or  asset  is  explicitly  assigned; 

•  inventory  accessibility  procedures  and  support;  and 

•  the  data  elements  required  for  each  inventory  item,  including 

•  cost  (e.g.,  history  of  actual  development  costs,  annual  operating  and 
maintenance  costs,  and  expected  life  cycle  costs)  of  each  item; 

•  owner  of  each  item; 

•  physical  location  of  each  item;  and 

•  the  logical  (e.g.,  architectural)  location  of  each  item. 

For  systems,  inventory  data  elements  could  be  part  of  the  organization's 
configuration  management  process.  They  could  also  include  schedule 
data,  such  as  dates  of  installation,  last  upgrade,  last  maintenance,  and  last 
security  patch.  For  personnel,  inventory  data  elements  could  include 
knowledge,  skills,  abilities,  salary,  and  last  performance  appraisal. 

Commitment  2:  An  official  is  assigned  responsibility  for  managing 
the  IT  asset  tracking  process. 

A  designated  official  is  necessary  to  adequately  manage  the  process.  The 
official  will  ensure  that  an  IT  asset  inventory  is  developed  and  maintained 
so  that  assets  are  accurately  tracked.  Staff  or  external  advisors  may  be 
assigned  to  assist  the  official  in  conducting  IT  asset  tracking  and  in 
verifying  and  validating  IT  asset  inventory  data. 

Prerequisite  1:  Adequate  resources  are  provided  for  performing 
the  IT  asset  tracking  activities. 

These  resources  typically  involve 

•  managerial  attention  to  the  process; 
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•  staff  support  including,  at  a  minimum,  a  designated  official  to  manage  the 
process;  and 

•  supporting  tools  and  equipment  for  tracking  IT  assets  which  may  include: 

•  an  inventory  database; 

•  inventory  reporting,  updating,  and  query  tools;  and 

•  a  method  for  communicating  inventory  changes  to  affected  parties. 

Prerequisite  2:  An  IT  investment  board  exists  and  oversees  the 
development  and  maintenance  of  IT  asset  tracking  activities. 

An  IT  investment  board 

•  authorizes  the  establishment  of  an  IT  asset  inventory  and  the  identification 
of  essential  data  elements  and  component  items, 

•  oversees  changes  to  the  IT  asset  inventory,  and 

•  oversees  the  on  demand  requests  and  notification  of  parties  using  the  IT 
asset  inventory-which  may  include  the  investment  board  members, 
accounting  or  finance  groups,  business  units,  and  the  budget  office. 

(See  Stage  2- IT  Investment  Board  Operation  for  a  description  of  the  roles 
and  responsibilities  of  each  investment  board.) 

Activities  Activity  1:  The  organization's  IT  asset  inventory  is  developed  and 

maintained  according  to  a  written  procedure. 

A  standard,  documented  procedure  is  used  so  that  developing  and 
maintaining  the  inventory  is  a  repeatable  event,  which  produces  inventory 
data  that  are  timely,  sufficient,  complete,  and  comparable.  The  inventory 
can  be  prepared  by  the  IS  support  component  of  an  organization  with  the 
verification  and  validation  performed  by  the  designated  official. 

An  IT  asset  inventory  typically  includes 

•  hardware  (e.g.,  computers,  monitors,  printers,  storage  devices, 
telecommunication  devices,  cables); 

•  software  (e.g.,  operating  systems,  databases,  applications); 
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Evidence  of  Performance 


•  personnel  (e.g.,  development  staff,  training  and  support  staff,  operations 
and  maintenance  (O&M)  staff); 

•  professional  services  (e.g.,  development  and  support  contracts,  leasing 
contracts,  service  contracts);  and 

•  software  licensing  agreements. 

Activity  2:  IT  asset  inventory  changes  are  maintained  according  to 
a  written  procedure. 

Changes  and  updates  to  the  inventory  are  maintained  in  an  orderly, 
documented  manner.  Maintaining  the  integrity  of  the  inventory  is 
important  to  ensure  that  the  inventory  is  a  useful  decision-making  tool. 
Proper  maintenance  procedures  may  be  indicated  by 

•  the  designation  of  an  individual  or  organizational  entity  responsible  for 
maintaining  the  integrity  of  the  inventory  and 

•  regularly  recorded  changes  and  updates. 

Activity  3:  Investment  information  is  available  on  demand  to 
decisionmakers  and  other  affected  parties. 

The  IT  inventory  is  only  of  value  to  the  extent  that  decisionmakers  and 
stakeholders  can  and  do  use  it.  Knowledge  of  the  contents  of  the  inventory 
by  staff  and  managers  throughout  the  organization  can  avoid  asset 
duplication  and  reconcile  overlapping  resources.  For  example,  an 
inventory  report  can  be  used  to  better  manage  the  licensing  of  an 
organization's  application  software  by  showing  individually  licensed 
applications  that  may  be  candidates  for  group  licensing. 

Activity  4:  Historical  IT  asset  inventory  records  are  maintained  for 
future  selections  and  assessments. 

The  inventory,  as  it  is  maintained,  becomes  an  archival  source  of 
information  that  can  be  used  during  future  project  selections  and 
investment  evaluations. 

E  vidence  1:  Physical  evidence  of  IT  Asset  T racking  exists. 

Physical  evidence  could  include,  for  example 

•  the  IT  asset  inventory; 
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•  inventory  database  tools  including  bar  coding  tags  and  detectors;  and 

•  inventory  reporting,  updating,  and  query  tools. 

Evidence  2:  Documentary  evidence  of  IT  Asset  T racking  is  created 
and  maintained. 

Documentary  evidence  could  include,  for  example 

•  written  policy  and  procedures  for  developing  and  maintaining  an  IT  asset 
inventory; 

•  an  IT  asset  inventory  report  including  records  for  categories  such  as 
hardware,  software,  personnel,  professional  services,  and  software 
licensing  agreements; 

•  records  of  IT  asset  inventory  changes  and  updates;  and 

•  inventory  reports  that  have  been  produced. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  IT  Asset  T racking. 

Testimonial  evidence  could  include,  for  example 

•  interviews  with  people  using  the  IT  asset  inventory  such  as  investment 
board  members,  accounting  or  finance  groups,  business  units,  and  the 
budget  office; 

•  inventory  staff  interviews;  and 

•  organization  staff  interviews. 


Page  51 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Section  5:  Critical  Processes  For  The  ITIM  Stages 
□  ITIM  Stage  2:  Building  the  Investment  Foundation 
ODD  Business  Needs  Identification  for  IT  Projects 


Business  Needs 
Identification  for  IT 
Projects 


The  benefits  of  IT  projects  and  investments  accrue  to  customers  or  end 
users  performing  an  organizational  business  process.  We  will  refer  to  both 
of  these  types  of  beneficiaries  as  users. 

This  critical  process  establishes  the  mechanism  for  identifying  the 
business  needs  and  the  associated  users  that  drive  each  IT  project.  Thus, 
this  critical  process  creates  the  link  between  the  organization's  business 
objectives  and  its  IT  strategy  and  creates  the  partnership  between  the 
benefiting  community  and  the  IT  solution  providers.  This  critical  process 
can  also  help  (1)  identify  the  sponsoring  executive(s)  or  organization(s) 
and  (2)  establish  the  performance  measures  (e.g.,  reducing  cycle  time, 
increasing  quality)  by  which  each  project's  benefits  will  be  assessed. 
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Figure  5.5:  Business  Needs  Identification  for  IT  Projects 
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Purpose 


Organizational  Commitment 


Prerequisites 


To  ensure  that  each  IT  project  supports  the  organization's 
business  needs  and  meets  users'  needs. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  identifying  the  business  needs  (and  the  associated 
users)  of  each  IT  project. 

The  organization  has  policies  and  procedures  that  outline  a  systematic 
process  for  identifying,  classifying,  and  organizing  its  business  needs  and 
the  IT  projects  used  to  support  these  needs.  In  many  cases,  this  can  be 
covered  in  internal  guidance  used  for  documenting  business  cases  for  IT 
investments. 

These  policies  and  procedures  typically  specify  that 

this  systematic  process  is  linked  to  the  business  planning  process, 

business  needs  or  opportunities  should  be  stated  in  functional  terms  or  in 
terms  of  desired  business  improvement  and  not  in  product-  or  technology- 
specific  terms, 

each  IT  project  fit  within  (or  be  waived  from)  the  organization's  enterprise 
IT  architecture, 

IT  projects  or  resources  that  do  not  support  an  identified  business  need 
(and  the  associated  customers  or  end  users)  are  further  examined  for 
possible  termination, 

the  procedure  by  which  similar  needs  or  opportunities  within  different 
operating  units  are  reconciled,  and 

business  needs  identification  occurs  regularly  as  part  of  the  strategic 
planning  cycle. 

Prerequisite  1:  Adequate  resources  are  provided  for  identifying 
business  needs  and  associated  users. 

These  resources  typically  involve 

funding  for  these  activities; 

managerial  attention  to  this  process; 

staff  support  for  carrying  out  these  activities;  and 
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•  supporting  methods,  analytical  tools,  and  processes. 

Prerequisite  2:  The  organization  has  defined  business  needs  or 
stated  mission  goals. 

These  mission  goals  and  business  needs  are  typically  identified  in: 

•  strategic  management  or  business  plans  (e.g.,  agency  strategic  plans 
prepared  for  GPRA), 

•  business  process  architecture  documents, 

•  process  improvement  initiatives,  or 

•  performance  measurement  plans. 

Defining  these  needs  or  goals,  however,  is  largely  outside  the  scope  of 
ITIM.  (See  also,  the  section  above  entitled,  Limitations  and  Boundaries  of 
ITIM.) 

Prerequisite  3:  IT  staff  are  trained  in  business  needs  identification. 

The  purpose  of  every  IT  project  is  to  support  identified  business  needs, 
opportunities,  or  mission  goals.  Thus,  the  IT  personnel  who  provide  these 
solutions  must  also  be  capable  of  understanding  the  business  needs  of 
their  end  users  and  the  organization's  work  process  customers.  Examples 
of  this  training  may  include 

•  relevant  conference  attendance, 

•  organizational  requirements  for  ongoing  education,  or 

•  staff  rotation  through  supported  business  units. 

Prerequisite  4:  All  IT  projects  are  identified  in  the  IT  asset 
inventory. 

To  ensure  that  each  IT  project  is  supporting  one  or  more  business  needs 
or  mission  goals,  the  organization  must  formally  identify  each  IT  project. 
This  identification  should  be  reflected  in  the  IT  Asset  Inventory. 

(See  also  Stage  2-IT  Asset  T racking  for  a  description  of  the  activities 
associated  with  developing  an  IT  asset  inventory.) 


Page  55 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Section  5:  Critical  Processes  For  The  ITIM  Stages 
□  ITIM  Stage  2:  Building  the  Investment  Foundation 
ODD  Business  Needs  Identification  for  IT  Projects 


Activities  Activity  1:  The  business  needs  for  each  IT  project  are  clearly 

identified  and  defined. 

Each  IT  project  is  directly  or  indirectly  linked  to  at  least  one  of  the 
organization's  business  needs  or  mission  goals,  with  a  direct  link  being  of 
greater  value  than  an  indirect  link.  This  link  can  be  established  in  a  variety 
of  ways.  For  example,  an  organization  can 

•  identify  a  project's  business  purpose  as  part  of  the  project's  initiation 
activities, 

•  define  an  executive  sponsor  for  each  project,  or 

•  obtain  validation  from  external  groups  supporting  the  business  value  of 
the  project. 

The  business  needs  for  each  IT  project  will  generally  be  documented  in 
the  business  case  for  the  project. 

Activity  2:  Specific  users  are  identified  for  each  IT  project. 

Every  IT  project  will  have  at  least  one  set  of  benefiting  end  users  or 
customers.  A  given  project  may  indeed  address  the  needs  of  multiple  sets 
of  end  users  or  customer  groups.  However,  the  identified  end  users  or 
customers  will  be  formally  identified  by  the  organization. 

Activity  3:  Identified  users  participate  in  project  management 
throughout  a  project's  life  cycle. 

Since  they  are  critical  to  each  IT  project's  success,  end  users  or  customers 
must  participate  in  each  project.  This  participation  can  involve 

•  being  identified  as  a  participant  in  an  integrated  project  management  team 
(e.g.,  a  multidisciplinary  team  comprised  of  business,  IT,  financial,  and 
other  members); 

•  using  a  project  charter  to  create  a  contract-like  relationship  between  the 
end  users  or  customers  and  the  IT  project  team; 

•  bringing  in  end  user  or  customer  representatives  to  participate  in 
intermediate  testing  of  the  system  being  developed; 

•  placing  customer  or  end  user  representatives  on  the  development  team; 


Page  56 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Evidence  of  Performance 


Section  5:  Critical  Processes  For  The  ITIM  Stages 
□  ITIM  Stage  2:  Building  the  Investment  Foundation 
ODD  Business  Needs  Identification  for  IT  Projects 


•  creating  an  internal  financial  charge-back  mechanism  for  IT  development 
projects;  or 

•  placing  the  IT  project  under  the  management  of  the  end  user's  or 
customer's  sponsor. 

Evidence  1:  Physical  evidence  of  Business  Needs  Identification  for 
IT  Projects  exists. 

Physical  evidence  could  include,  for  example, 

•  process  improvement  initiatives, 

•  training  classes, 

•  the  management  of  IT  project  efforts  by  its  sponsor, 

•  end  user/customer  participation  in  intermediate  testing, 

•  end  user/customer  membership  on  an  IT  project  development  team,  and 

•  an  internal  financial  charge-back  mechanism  for  IT  development  projects. 

Evidence  2:  Documentary  evidence  of  Business  Needs 
Identification  for  IT  Projects  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example, 

•  a  written  policy  and  process  (e.g.,  a  business  case  methodology)  for 
establishing  the  business  needs  of  each  IT  project; 

•  defined  business  processes  or  stated  mission  purposes  (documented  in  a 
strategic  management  or  business  plan  (e.g.,  agency  strategic  plans 
prepared  for  GPRA),  business  process  architecture  documents,  or 
performance  measurement  plans); 

•  contract-like  relationship  in  IT  project  charters;  and 

•  IT  project  membership  that  includes  end  users/customers. 
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Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
for  Business  Needs  Identification  of  IT  Projects. 

Testimonial  evidence  could  include,  for  example, 

•  training  instructor  interviews, 

•  IT  project  sponsor  interviews,  and 

•  end  user/customer  representative  interviews. 
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The  purpose  of  this  critical  process  is  to  establish  a  process  for  selecting 
IT  proposals.  A  proposal  selection  process  is  a  basic  step  toward 
implementing  a  mature  IT  proposal  and  project  selection  process  in  Stage 
3.  The  key  activities  implemented  within  this  process  are  (1)  concurrent 
review  of  IT  proposals  by  the  organization's  executives,  (2)  the  use  of 
predefined  selection  criteria  to  analyze  the  proposals,  and  (3)  structured 
decision-making  by  executives  to  fund  some  proposals  and  not  fund 
others. 
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Figure  5.6:  Proposal  Selection 
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Purpose 

Organizational  Commitment 


Prerequisites 


To  ensure  that  an  established,  structured  process  is  used  to  select 
new  IT  proposals. 

Commitment  1:  Executives  and  managers  follow  an  established 
selection  process. 

To  ensure  that  the  most  meritorious  proposals  within  an  organization  are 
selected  fairly,  executives  and  line  managers  must  accept  and  fully 
support  the  proposal  selection  process.  They  should  have  confidence  that 
their  proposals  will  be  objectively  assessed  as  a  result  of  the  process. 

Executives  must  understand  the  purpose  behind  the  proposal  selection 
process  and  be  able  to  competently  execute  their  duties  within  the 
process.  As  a  basic  step  toward  a  mature  selection  process,  this  critical 
process  can  be  performed  with  or  without  the  creation  of  an  IT  investment 
board. 

Commitment  2:  An  official  is  designated  to  manage  the  proposal 
selection  process. 

A  designated  official  is  necessary  to  adequately  manage  the  process, 
particularly  in  an  organization  with  little  or  no  experience  with  IT 
investment  management.  The  designated  official  will  ensure  that 

•  the  managers  and  staff  in  the  organization  are  aware  of  the  key  events  in 
the  process, 

•  proposals  are  collected  and  presented  uniformly  to  senior  management, 
and 

•  feedback  is  provided  to  the  affected  parties. 

Prerequisite  1:  Adequate  resources  are  provided  for  proposal 
selection  activities. 

These  resources  typically  involve 

•  managerial  time  and  attention  to  the  process; 

•  staff  support  including,  at  a  minimum,  a  designated  official  to  manage  the 
process;  and 

•  supporting  tools,  methods,  and  equipment  for  organizing  and  analyzing  the 
proposals. 


Page  62 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Activities 


Evidence  of  Performance 


Section  5:  Critical  Processes  For  The  ITIM  Stages 
□  ITIM  Stage  2:  Building  the  Investment  Foundation 
□□□  Proposal  Selection 


Activity  1:  The  organization  uses  a  structured  process  to  develop 
new  IT  proposals. 

The  organization  will  use  an  organized,  structured  process  for  requesting 
IT  proposals  that  require  funding  or  organizational  support.  This  activity 
will  typically  occur  within  the  context  of  the  organization's  cyclical 
budgeting  process.  A  designated  official  will  manage  the  data  submission 
and  screening  activities  associated  with  the  process. 

Activity  2:  Executives  analyze  and  prioritize  new  IT  proposals 
according  to  established  selection  criteria. 

Executives  receive  and  compare  the  submitted  proposals  to  one  another 
using  previously  agreed  upon  selection  criteria  such  as  cost  and  schedule. 
They  should  generally  incorporate  some  type  of  cost/benefit  analysis  with 
respect  to  the  strategic  goals  and  missions  of  the  organization.  (These 
selection  criteria  serve  as  the  starting  point  for  the  development  of  the 
more  mature  decision-making  criteria  in  Stage  3.)  Once  the  proposals  have 
been  compared  using  the  selection  criteria,  the  executives  prioritize  the 
new  proposals. 

Activity  3:  Executives  make  funding  decisions  for  new  IT  proposals 
according  to  an  established  process. 

The  organization's  executives  have  discretion  in  making  the  final  funding 
decisions  on  IT  proposals.  However,  their  decisions  should  be  based  upon 
the  analysis  that  took  place  in  the  previous  activities.  Additionally,  there 
should  be  evidence  that  some  proposals  are  judged  less  meritorious  than 
others  and  thus  do  not  get  funded  as  part  of  the  decision-making  process. 

Evidence  1:  Physical  evidence  of  Proposal  Selection  exists. 

Physical  evidence  could  include,  for  example 

•  the  identification  of  a  proposal  selection  process  manager, 

•  new  proposals  being  collected  and  sent  to  the  IT  board, 

•  proposal  comparison  and  prioritization  by  the  IT  board,  and 

•  a  board  consensus  on  final  funding  decisions. 

Evidence  2:  Documentary  evidence  of  Proposal  Selection  is  created 
and  maintained. 
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Documentary  evidence  could  include,  for  example: 

•  a  documented  proposal  selection  process, 

•  a  proposal  solicitation  memorandum,  and 

•  final  funding  decision  memorandum. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Proposal  Selection. 

Testimonial  evidence  could  include,  for  example: 

•  board  member  interviews, 

•  proposal  team  interviews,  and 

•  proposal  selection  manager  interviews. 
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ITIM  Stage  3: 
Developing  a  Complete 
Investment  Portfolio 


Maturity  Stages 


Critical  Processes 


□  Investment  Process  Benchmarking 
IT-Driven  Strategic  Business  Change 


^Post-Implementation  Reviews  and  Feedback 
^Portfolio  Performance  Evaluation  and  Improvement 
Systems  and  Technology  Succession  Management 

Authority  Alignment  of  IT  Investment  Boards 

□  Portfolio  Selection  Criteria  Definition 

□  Investment  Analysis 

□  Portfolio  Development 

□  Portfolio  Performance  Oversight 


□  IT  Investment  Board  Operation 
HlT  Project  Oversight 

□  IT  Asset  Tracking 

Business  Needs  Identification  for  IT  Projects 

□  Proposal  Selection 

IT  Spending  without  Disciplined  Investment 
Processes 


During  Stage  3,  the  IT  investment  board  enhances  the  IT  investment 
management  process  by  developing  a  complete  investment  portfolio. 
Taking  a  portfolio  perspective  enables  the  organization  to  consider  its 
investments  in  a  comprehensive  manner  so  that  the  investments  address 
the  strategic  goals,  objectives,  and  mission  of  the  organization.  The 
organization  develops  its  IT  investment  portfolio  by  combining  all  IT 
assets,  resources,  and  investments  owned  by  an  organization,  considering 
new  proposals  along  with  previously  funded  investments  and  identifying 
the  appropriate  mix  of  IT  investments  that  best  meet  its  mission  needs  and 
improvement  priorities.  As  such,  this  maturity  stage  is  comprised  of  the 
following  five  critical  processes: 

•  Authority  Alignment  of  IT  I  nvestment  Boards  is  the  process  for 
coordinating  the  responsibilities  and  activities  of  the  IT  investment  boards 
when  an  organization  uses  multiple  boards. 

Criteria:  Information  Technology  investment  (AIM  D-96-64),  p.  25;  IT 
Assessment  Guide  ( AIMD-10.1.13),  p.  9-10. 
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•  Portfolio  Selection  Criteria  Definition  is  the  process  used  by 
decisionmakers  to  create  and  communicate  to  the  organization  the  criteria 
used  to  select  and  fund  IT  investments. 

Criteria:  IT  Assessment  Guide  (AIM  D-10.1,13),  p.  27-29,  45-46  (CCA);  OMB 
IT  Investment  Guide,  p.  7-9. 

•  Investment  Analysis  is  the  process  for  examining  the  fundamental  cost, 
benefit,  schedule,  and  risk  characteristics  of  each  IT  investment  before 
they  are  funded  and  combined  with  other  investments  into  a  portfolio. 

Criteria:  IT  Assessment  Guide  ( AIMD-10.1.13),  p.  52,  (CCA,  OMB  A-94, 

OMB  A-130,  OM  B  M -97-0(2));  OMB  IT  Investment  Guide,  p.  6-7. 

•  Portfolio  Development  is  the  process  for  comparing  worthwhile 
investments  and  then  combining  selected  investments  into  a  funded 
portfolio. 

Criteria:  IT  Assessment  Guide  (AIMD-10.1.13),  p.  32-35,  Capital 
Programming  Guide,  p.  16-17;  (CCA,  OMB  M -97-0(2)). 

•  Portfolio  Performance  Management  is  a  process  that  builds  upon  the 
Stage  2  IT  Project  Oversight  critical  process  by  adding  the  elements  of 
investment  benefit  and  risk  management  to  the  control  process  activities. 

Criteria:  IT  Assessment  Guide  ( AIMD-10.1.13),  p.  52-55,  (CCA,  PRA,  FASA, 
EO  13011,0MB  A-ll,  Part  3);  Information  Technology  Investment  [  AIMD- 
96-64),  p.  65;  IT  Assessment  Guide  (AIM  D-10. 1.13),  p.  61-62,  (CCA,GPRA, 
CFO,  OMB  A-127,  OMB  A-123). 
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Authority  Alignment  of  IT 
Investment  Boards 


IT  proposals  and  projects  may  originate  from  any  organizational  unit. 
Using  decision  parameters  such  as  funding  level  requirements,  degree  of 
risk,  type  of  investment,  and  organizational  scope  of  expected  benefits 
along  with  defining  each  IT  investment  board's  scope  of  responsibility 
ensures  that  the  organization  selects  IT  proposals  and  projects  as 
investments  at  the  appropriate  organizational  level  (e.g.,  corporate, 
division,  office),  by  the  appropriate  IT  investment  board.  For  example, 
purchasing  a  dozen  desktop  computers  for  a  specific  work  unit  generally 
would  not  warrant  the  involvement  of  top  executives.  Conversely, 
designing  or  buying  new  software  applications  for  organizationwide 
financial  management  activities  typically  requires  broad  executive 
sponsorship,  participation,  and  review. 


This  critical  process  is  based  on  the  assumption  that  the  organization  has, 
or  in  the  future  might  have,  more  than  one  IT  investment  board.  For 
example,  an  organization  may  create  one  enterprisewide  board  to  make 
strategic  or  organizationwide  IT  decisions  and  also  may  create  lower  level 
boards  to  make  IT  decisions  unique  to  specific  business  entities.  If  the 
organization  has  only  one  IT  investment  board,  then  the  common  features, 
practices,  and  activities  in  this  critical  process  should  be  implicitly 
executed  by  this  single  board. 
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Figure  5.7:  Authority  Alignment  of  IT  Investment  Boards 
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Purpose 

Organizational  Commitment 


T o  ensure  that  IT  investments  are  selected  and  managed  by  the 
appropriate  investment  board. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  aligning  IT  decision-making  authority  among  the 
existing  boards. 

The  organization  has  documented  policies  and  procedures  that  describe 
the  processes  for  aligning  and  coordinating  IT  investment  decision-making 
assignment  and  authority  and  for  modifying  these  processes  later. 

These  policies  and  procedures  should  specify  the  criteria  for  determining 
where  in  the  organization  different  types  of  IT  investment  decisions  are 
made.  Typically,  they  should  specify 

•  that  individual  business  or  operational  units  should  retain  decision-making 
authority  for  unit  specific  IT  decisions  (while  still  following 
enterprisewide  standards  and  procedures); 

•  the  relationship  of  the  IT  boards  to  the  organization's  enterprise  IT 
architecture; 

•  that  critical  infrastructure  investments  and  proposals  (e.g., 
telecommunications,  networks,  large  scale  data  processing)  should  usually 
be  centrally  controlled  and  monitored  by  the  enterprisewide  IT  investment 
board; 

•  that  cross-functional  investments  and  proposals  (e.g.,  organizationwide 
common  applications)  which  could  affect  many  departments  and  users 
across  the  organization  should  be  raised  to  the  enterprisewide  IT 
investment  board  to  ensure  that  the  managers  and  users  across  these 
various  departments  buy  into  these  investments  and  proposals; 

•  that  IT  investments  and  proposals  with  high  cost  or  high  risk  or  significant 
scope  and  duration  should  be  considered  by  the  enterprisewide  IT 
investment  board;  and 

•  the  procedure  for  passing  decision-making  assignment  and  authority  for  a 
given  investment  or  proposal  from  one  board  to  another. 

Commitment  2:  One  enterprisewide  IT  investment  board  exists 
within  an  organization  and  this  board  must  be  capable  of  making 
the  final  investment  decisions. 
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Regardless  of  the  decision-making  coordination  mechanisms  or  hierarchy 
used  by  the  organization,  there  can  be  only  one  enterprise  level  IT 
investment  board  and  it  must  be  capable  of  reviewing  lower  level  board 
actions  and  invoking  final  decision-making  authority  over  all  IT 
investments.  If  disputes  or  disagreements  arise  over  decision-making 
jurisdiction  about  a  specific  IT  investment  project,  the  enterprisewide 
board  must  be  able  to  resolve  the  issue  (i.e.,  by  mediating  disputes  among 
unit-specific  boards). 

Prerequisites  Prerequisite  1:  Adequate  resources  are  provided  for  decision¬ 

making  authority  alignment  activities. 

These  resources  typically  involve 

•  the  attention  and  support  of  the  managers  and  executives  involved  in  the 
process  and 

•  staff  support  for  executing  the  activities. 

Prerequisite  2:  A  working  group  is  designated  to  be  responsible  for 
creating  and  maintaining  decision-making  authority  alignment 
policies  and  procedures. 

A  designated  working  group-generally  comprised  of  individuals  from 
different  parts  of  the  organization-is  identified  and  chartered  with  this 
responsibility.  This  working  group  will  formulate  the  policies  and 
procedures  and  senior  management  will  then  typically  ratify  the  policies 
and  procedures.  This  group  is  also  expected  to  maintain  and  modify  these 
policies  and  procedures  as  needed  in  consultation  with  and  with  the 
approval  of  senior  executives. 

Activities  Activity  1:  Criteria  for  aligning  IT  investment  decision-making 

authority  are  established  and  maintained. 

The  alignment  criteria  are  used  by  the  IT  investment  boards  to  redistribute 
IT  investment  assignment,  decision-making,  and  review  authority  within 
the  organization.  These  criteria  can  be  based  on  cost,  benefit,  schedule, 
and  risk  (CBSR)  thresholds,  the  number  of  users  affected,  business  unit 
function  (e.g.,  CIO,  human  resources,  or  program  office),  life  cycle  phase 
(e.g.,  research  and  development  [R&D],  full  scale  development,  or  O&M), 
or  other  comparable  and  useful  measures.  For  example,  an  organization 
might  decide  that  investments  with  less  than  a  $100,000  lifecycle  cost 
should  be  managed  at  the  lowest  departmental  level,  while  investments 
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with  more  than  $100  million  in  lifecycle  cost  should  be  managed  by  the 
enterprisewide  investment  board. 

Activity  2:  Each  IT  investment  or  proposal  is  considered  by  an  IT 
investment  board  based  upon  the  alignment  criteria. 

Based  on  the  policy,  criteria,  and  thresholds,  decision-making  authority  for 
each  IT  investment  or  proposal  is  considered  by  a  specific  board  at  the 
appropriate  organizational  level.  Once  an  IT  investment  is  assigned  to  an 
investment  board,  it  generally  should  not  be  reassigned.  This  promotes 
investment  board  ownership,  investment  responsibility,  and  the  retention 
of  an  investment  history. 

Activity  3:  Each  IT  investment  board  operates  in  accordance  with 
its  assigned  scope  and  area  of  authority. 

For  the  whole  IT  investment  management  process  to  function  smoothly 
and  effectively,  each  investment  board  must  understand  the  established 
policies  and  procedures  of  the  IT  investment  decision-making  authority 
alignment  and  execute  them  within  their  scope  and  area  of  authority. 

Evidence  1:  Physical  evidence  of  Authority  Alignment  of  IT 
Investment  Boards  exists. 

Physical  evidence  could  include,  for  example 

•  each  IT  investment  board  executing  the  decision-making  authority 
alignment  policy  and 

•  the  existence  of  an  authority  alignment  policy  working  group. 

Evidence  2:  Documentary  evidence  of  Authority  Alignment  of  IT 
Investment  Boards  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example 

•  a  written  policy  for  decision-making  authority  alignment, 

•  specific  criteria  and  thresholds  for  coordinating  IT  investment  authority 
alignment,  and 

•  IT  board  assignment  memoranda/documents. 
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Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Authority  Alignment  of  IT  Investment  Boards. 

Testimonial  evidence  could  include,  for  example 

policy  working  group  interviews  and 

investment  review  board  interviews. 
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Portfolio  Selection  Criteria 
Definition 


Portfolio  selection  criteria  are  a  necessary  part  of  an  IT  investment 
management  process.  Developing  an  IT  investment  portfolio  involves 
defining  appropriate  IT  investment  cost,  benefit,  schedule,  and  risk 
(CBSR)  criteria  to  ensure  that  the  organization's  strategic  goals, 
objectives,  and  mission  will  be  satisfied  by  the  selected  investments. 
Portfolio  selection  criteria  reflect  the  strategic  and  enterprisewide  focus  of 
the  organization  and  may  be  different  from  criteria  used  to  select 
individual  IT  projects.  When  IT  projects  are  not  considered  in  the  context 
of  a  portfolio,  criteria  based  on  narrow,  lower-level  requirements  may 
dominate  organizationwide  selection  criteria.  IT  projects  sometimes  are 
selected  on  the  basis  of  an  isolated  business  need,  the  type  and  availability 
of  funds,  or  the  receptivity  of  management  to  a  project  proposal.  The 
portfolio  selection  criteria  are  used  by  the  organization's  IT  investment 
board  to  select  the  IT  investments  that  best  support  the  organization's 
mission  and  clearly  communicate  to  project  managers  the  investment 
board's  selection  priorities.  Additional  criteria  should  address  alignment 
with  mission  needs,  organizational  strategy,  and  line-of-business  priorities. 
If  an  organization's  mission  or  business  needs  and  strategies  change,  these 
criteria  should  be  re-examined.  These  criteria  should  also  be  applied  as 
uniformly  as  possible  throughout  the  organization  to  ensure  decision¬ 
making  consistency  and  process  institutionalization. 
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Figure  5.8:  Portfolio  Selection  Criteria  Definition 


Page  75 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Section  5:  Critical  Processes  For  The  ITIM  Stages 
□  ITI M  Stage  3:  Developing  a  Complete  Investment  Portfolio 
ODD  Portfolio  Selection  Criteria  Definition 


Purpose 


Organizational  Commitment 


Prerequisites 


To  ensure  that  the  organization  develops  and  maintains  IT 
portfolio  selection  criteria  that  support  its  mission,  organizational 
strategies,  and  business  priorities. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  creating  and  modifying  IT  portfolio  selection 
criteria. 

The  organization  has  policies  and  procedures  that  outline  a  systematic 
process  for  creating  and  modifying  the  selection  criteria. 

These  policies  and  procedures  typically  specify 

a  working  group,  or  person,  that  is  designated  to  manage  the  criteria 
creation  and  modification  process; 

the  link  to  the  organization's  strategic  plans,  budget  processes,  and 
enterprise  IT  architecture; 

the  key  information  elements  required  to  create  or  modify  the  selection 
criteria; 

suggested  investment  and  proposal  selection  criteria;  and 

to  whom  the  selection  criteria  should  be  distributed. 

Prerequisite  1:  Adequate  resources  are  provided  for  selection 
criteria  definition  activities. 

These  resources  typically  involve 

the  time  and  attention  of  the  executives  involved  in  the  process, 
staff  to  support  the  activities  within  this  process,  and 
supporting  tools  and  equipment. 

Prerequisite  2:  A  working  group  is  designated  to  be  responsible  for 
creating  and  modifying  the  IT  portfolio  selection  criteria. 

A  group  of  people  is  designated  to  be  responsible  for  managing  the 
creation  and  modification  of  the  selection  criteria.  This  group  should 
incorporate  the  organization's  mission,  strategy,  and  priorities  into  the 
criteria.  Thus,  this  group  might  be  the  IT  investment  board,  a  subset  of  this 
board  to  include  the  CIO  or  some  other  executive  management  team. 
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Activities  Activity  1:  The  enterprisewide  IT  investment  board  approves  the 

core  IT  portfolio  selection  criteria,  including  C BSR  criteria,  based 
on  the  organization's  mission,  goals,  strategies,  and  priorities. 

The  selection  criteria  should  be  linked  directly  to  the  organization's 
broader  mission,  goals,  strategies,  and  priorities.  This  ensures  that  the 
selected  IT  investments  will  support  these  larger  organizational  tenets  and 
purposes.  It  is  very  important  that  the  criteria  also  take  into  account  the 
organization's  IT  architecture  so  as  to  (1)  avoid  unwarranted  overlap 
across  investments  and  (2)  ensure  maximum  systems  interoperability. 

The  selection  criteria  used  for  assessing  and  ranking  individual 
investments  and  proposals  should  generally  consist  of  the  four  essential 
investment  elements:  costs,  benefits,  schedule,  and  risks.  Organizations 
typically  establish  broad  categories  related  to  the  following  four  areas  and 
then  develop  more  specific  sub-elements  under  each  broad  category. 

•  Cost  may  include  lifecycle  costs  broken  apart  into  initial  costs,  ongoing 
development  costs,  and  indirect  costs. 

•  Benefit  may  include  tangible  benefits  and  intangible  benefits  constructed 
using  a  variety  of  techniques  (cost/benefit  analyses  using  net  present 
value,  return  on  investment  calculations). 

•  Schedule  could  include  the  lifecycle  schedule  and  the  schedule  of  benefits. 

•  Risk  can  include  investment,  organizational,  funding,  and  technical  risks. 

The  organization  must  determine  how  these  criteria  are  applied  and  used 
to  select  IT  investments  for  the  portfolio.  Costs  and  benefits  are  both 
affected  by  risks.  A  risk-adjusted  return  on  investment  type  of  calculation 
could  combine  all  of  these  categories.  The  selection  criteria  also  may 
include  a  description  of  an  investment's  or  proposal's  minimum  or 
maximum  acceptable  CBSR  thresholds  (e.g.,  a  minimum  acceptable  return 
on  investment  hurdle  rate  or  a  maximum  acceptable  schedule  length). 

An  organization  could  use  a  weighting  schema  when  creating  the  selection 
criteria.  The  organization  should  assign  weights  to  each  of  the  broad 
categories,  as  well  as  any  sub-elements  related  to  each  category.  This  is 
done  to  help  prioritize  those  sub-elements  that  the  organization  considers 
the  most  significant  (e.g.,  an  organization  that  has  limited  experience 
developing  systems  may  give  technical  risk  a  greater  weight  than  projected 
cost).  Alternatively,  other  risk  analysis  methods  might  incorporate  the 
same  "weighting"  effect. 
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Evidence  of  Performance 


The  mixture  of  weights  among  the  ranking  criteria  will  vary  from 
organization  to  organization.  The  weighting  schema  used  should  take  into 
account  the  agency's  unique  mission,  capabilities,  and  limitations.  The 
organization  may  also  create  different  weighting  schemas  for  different 
kinds  of  investments  (e.g.,  operational,  infrastructure,  applications 
development  investments,  R&D).  These  weights  may  need  to  be  refined 
over  time  as  the  organization  gains  more  operational  experience  using  the 
weighting  schema.  Additionally,  as  a  starting  point,  the  organization  may 
want  to  borrow  selection  criteria  used  by  other  comparable  organizations. 

Activity  2:  The  IT  portfolio  selection  criteria  are  distributed 
throughout  the  organization. 

The  criteria  should  be  distributed  to  each  IT  investment  board  and  all  of 
the  IT  project  managers,  organizational  planners,  and  any  other  interested 
parties.  The  selection  criteria  should  be  clearly  addressed  in  IT  project 
funding  submissions. 

In  a  larger  organization  with  multiple  IT  investment  boards,  a  lower  level 
board  may  add  its  own  criteria  to  these  selection  criteria.  However,  it 
cannot  reduce  this  core  selection  criterion  set. 

(See  Stage  3— Authority  Alignment  of  IT  I  nvestment  Boards  for  the 
activities  associated  with  coordinating  the  responsibilities  of  multiple  IT 
investment  boards.) 

Activity  3:  The  IT  portfolio  selection  criteria  are  reviewed  using 
cumulative  experience  and  event-driven  data  and  modified,  as 
appropriate. 

The  IT  investment  selection  criteria  occasionally  may  be  changed  based  on 
(1)  historical  experience,  (2)  changes  in  the  organization's  strategic 
direction,  business  goals  or  priorities,  or  (3)  other  factors,  such  as 
increased  IT  management  capabilities  or  technological  changes. 

Ultimately,  however,  the  task  of  modifying  the  criteria  will  be  based  on  the 
experience  and  judgment  of  the  enterprisewide  investment  board. 

Evidence  1:  Physical  evidence  of  Portfolio  Selection  Criteria 
Definition  exists. 

Physical  evidence  could  include,  for  example 
•  an  IT  investment  scoring  model  or  decision  support  tool, 
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•  meetings  of  the  working  group  managing  creation  of  portfolio  selection 
criteria,  and 

•  possession  and  use  of  IT  portfolio  selection  criteria  by  project  managers. 

Evidence  2:  Documentary  evidence  of  Portfolio  Selection  Criteria 
Definition  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example 

•  the  written  policy  for  creating  and  modifying  the  IT  portfolio  selection 
criteria; 

•  the  IT  portfolio  selection  criteria  (including  cost,  schedule,  benefit,  and 
risk  elements); 

•  the  selection  criteria  weighting  schema; 

•  a  distribution  list  for  selection  criteria;  and 

•  evidence  of  historic  experience  using  selection  criteria. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Portfolio  Selection  Criteria  Definition. 

Testimonial  evidence  could  include,  for  example 

•  working  group  interviews  and 

•  project  manager  interviews. 
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Investment  Analysis 


IT  investment  analysis  is  one  of  the  basic  building  blocks  of  the  IT 
investment  management  approach.  Many  IT  investments  are  a  form  of 
capital  investment  used  by  an  organization  to  improve  business 
performance,  achieve  its  mission  goals,  and  satisfy  customers.  This  critical 
process  establishes  the  mechanism  for  (1)  analyzing  each  investment 
based  on  its  expected  costs,  benefits,  schedule  and  risks,  (2)  comparing 
each  investment  against  the  organization's  portfolio  selection  criteria,  and 
(3)  creating  a  prioritized  list  of  investments  that  align  with  mission 
improvement  goals  and  organizational  direction.  With  this  critical 
process,  the  organization  focuses  on  benefit  measurement  and  risk 
management  as  significant  factors  with  each  investment. 
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Figure  5.9:  Investment  Analysis 
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To  ensure  that  all  IT  investments  are  consistently  analyzed  and 
prioritized  according  to  the  organization's  portfolio  selection 
criteria. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  analyzing  IT  investments. 

The  organization  has  documented  policies  and  procedures  that  describe 
the  processes  for  analyzing  IT  investments  under  consideration.  These 
policies  and  procedures  typically  specify  the  following: 

•  Each  IT  investment  board  must  use  investment  analysis  as  the  basis  for 
decision-making  to  ensure  that  (1)  the  investments  that  will  best  serve  the 
organization  are  selected  and  (2)  all  investments  are  fairly  considered. 

•  Key  tasks  that  decisionmakers  should  consider  when  analyzing 
investments  relative  to  the  IT  portfolio  selection  criteria.  For  example,  the 
scoring  process-including  explanations  and  definitions  for  scores-should 
be  documented.  This  is  important  for  composite  scoring  methodologies 
used  for  rating  investments  on  any  or  all  of  the  CBSR  criteria. 

•  Investment  projects  with  missing  or  invalid  project  management  data  will 
be  corrected  or  removed  from  funding  consideration. 

•  Decisionmakers  should  identify  and  address  IT  investments  and  proposals 
that  are  conflicting,  overlapping,  strategically  unlinked,  or  redundant.  To 
help  do  this,  the  IT  investment  board  should  take  into  consideration  the 
organization's  IT  architecture. 

•  Initial  investment  and  proposal  information  screening  mechanisms  should 
be  considered  by  larger  organizations  or  organizations  with  many  IT 
investments  and  proposals. 

•  Investments  should  be  "modularized"  (e.g.,  managed  and  procured  in  well- 
defined  useful  segments  or  "modules"  that  are  short  in  duration  and  small 
in  scope)  to  the  maximum  extent  achievable. 

Prerequisite  1:  Adequate  resources  are  provided  for  investment 
analysis  activities. 

These  resources  typically  involve 

•  managerial  time  and  attention  to  focus  on  investment  analysis, 
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•  staff  support  for  carrying  out  activities  within  this  critical  process,  and 

•  supporting  tools  and  equipment  to  be  used  by  the  staff. 

Prerequisite  2:  IT  investment  portfolio  selection  criteria  have  been 
developed. 

The  Stage  3-Portfolio  Selection  Criteria  Definition  critical  process  must  be 
implemented  before  this  critical  process  can  be  accomplished. 

Prerequisite  3:  Information  from  the  IT  asset  inventory  is  used  by 
the  IT  investment  board. 

The  asset  inventory  is  necessary  to  ensure  that  each  board  is  aware  of  all 
of  the  investments  and  resources  for  which  it  is  responsible. 

(See  also  Stage  2-IT  Asset  T racking  for  a  description  of  the  activities 
associated  with  developing  an  IT  asset  inventory). 

Activities  Activity  1:  Each  IT  investment  board  ensures  that  the  CBSR  data 

and  other  required  data  are  validated  for  each  investment  within 
its  span  of  control. 

The  cognizant  IT  investment  board  is  responsible  for  ensuring  that  the 
data  submitted  by  each  IT  investment  and  IT  proposal  are  valid.  Once 
validated,  these  data  will  typically  become  the  baseline  performance 
expectations  against  which  the  investment  will  be  measured  in  the  future. 
Validation  activities  typically  include 

•  ensuring  the  completeness,  timeliness,  and  accuracy  of  the  data; 

•  investigating  the  assumptions  on  which  the  investment  is  based; 

•  understanding  the  sensitivity  of  CBSR  estimates  to  potential  changes  or 
disruptions  (e.g.,  the  risk  of  investment  delay  due  to  unseen  significant 
demand  for  the  manufacturer's  hardware); 

•  receiving  signed  memoranda  or  meeting  with  potential  end  users, 
customers,  and  sponsors  to  assess  support  for  the  investment  and  identify 
similar  external  efforts  (a  potential  verification  and  validation  source); 

•  analyzing  links  between  the  investment  and  the  organization's  mission, 
strategies,  and  plans;  and 

•  ensuring  that  the  investment  is  not  duplicative. 
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Though  ultimately  responsible  for  the  results,  IT  investment  boards  in 
larger  organizations  may  delegate  much  of  this  activity  to  supporting  staff 
or  working  groups.  For  large,  long-term,  expensive,  or  important 
investments,  the  board  may  also  have  internal  audit  and  quality  control 
staff  or  external  reviewers  and  experts  analyze  and  comment  on  the 
validity  of  the  investments'  CBSR  data. 

Activity  2:  Each  IT  investment  board  assesses  each  of  its  IT 
investments  with  respect  to  the  IT  portfolio  selection  criteria. 

Using  the  validated  investment  or  proposal  data,  the  board  assesses  and 
reviews  each  individual  investment  relative  to  the  established  selection 
criteria.  Depending  on  the  size  of  the  organization,  supporting  staff  or 
working  groups  may  first  assess  the  investments  using  the  criteria  or  the 
executives  themselves  may  assess  the  investments.  The  output  from  this 
activity  should  be  a  list  of  the  investments  and  their  associated 
assessments. 

One  technique  an  organization  may  consider  is  the  use  of  scoring  to 
evaluate  IT  investments.  With  a  scoring  technique,  the  assessment  body 
typically  attaches  numerical  scores  and  "relative  value"  weights  to  each  of 
the  individual  selection  criteria.  I  nvestments  are  then  assessed  relative  to 
these  scores  and  then  against  weights  associated  with  each  individual 
criterion.  F  inally,  the  weighted  scores  are  summed  to  create  a  numerical 
value  for  each  investment.  These  scoring  exercises,  however,  should  not 
be  used  as  surrogates  for  a  thorough  analysis  and  review  of  CBSR  data. 

Activity  3:  Each  IT  investment  board  prioritizes  its  full  portfolio  of 
IT  investments  using  the  portfolio  selection  criteria. 

Based  on  the  output  from  the  preceding  assessment  activity,  the  board 
creates  a  ranked  list  of  all  proposed  and  ongoing  IT  investments.  This  list 
will  provide  the  basis  for  decision-making  in  the  Portfolio  Development 
critical  process. 

E  vidence  1:  Physical  evidence  of  I  nvestment  Analysis  exists. 

Physical  evidence  could  include,  for  example 

•  examples  of  each  IT  board  validating  CBSR  data  for  each  investment; 

•  meetings  with  project  staff,  end  users,  customers,  and  sponsors; 

•  each  IT  board  analyzing  and  scoring  each  IT  investment;  and 
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•  each  IT  board  prioritizing  each  IT  investment. 

Evidence  2:  Documentary  evidence  of  Investment  Analysis  is 
created  and  maintained. 

Documentary  evidence  could  include,  for  example, 

•  a  written  policy  and  procedure  for  analyzing  IT  investments, 

•  an  investment  scoring  method  (especially  for  composite  scoring  methods), 

•  IT  investment  validation  data, 

•  a  list  of  IT  investments  and  associated  scores,  and 

•  a  prioritized  ranked  list  of  IT  investments. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Investment  Analysis. 

Testimonial  evidence  could  include,  for  example, 

•  IT  board  interviews  and 

•  working  group  interviews. 
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Individual  IT  investments  vary  in  type  and  purpose.  Some  investments  may 
involve  purchasing  hardware,  others  involve  developing  software,  and  still 
others  may  involve  operating  or  maintaining  IT  systems.  The  portfolio 
development  process  ensures  that  each  IT  investment  board  collectively 
analyzes  and  compares  all  investments  and  proposals  to  select  those  that 
best  fit  with  the  strategic  business  direction,  needs,  and  priorities  of  the 
organization. 

Additionally,  each  organization  has  practical  limits  on  funding,  the  risks  it 
is  willing  to  take,  and  the  length  of  time  for  which  it  will  incur  costs  on  a 
given  investment  before  benefits  are  realized.  To  address  these  practical 
limits,  the  portfolio  development  process  primarily  uses  categorization  to 
aid  in  investment  comparability  and  CBSR  oversight.  Categorization 
involves  grouping  investments  and  proposals  into  predefined  logical 
categories.  Once  this  is  accomplished,  investments  and  proposals  can  be 
compared  to  one  another  within  and  across  the  portfolio  categories  and 
the  best  overall  portfolio  can  then  be  selected  for  funding. 
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Figure  5.10:  Portfolio  Development 
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Purpose 


Organizational  Commitment 


Prerequisites 


To  ensure  that  an  optimal  IT  investment  portfolio  with  manageable 
risks  and  returns  is  selected  and  funded. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  establishing  and  maintaining  the  portfolio 
development  process. 

The  organization's  policies  and  procedures  for  developing  IT  investment 
portfolios  typically 

provide  common  definitions  for  IT  investment  portfolio  categories, 

apply  to  each  IT  investment  board  as  each  develops  it  comprehensive  IT 
investment  portfolio, 

stipulate  conditions  that  must  be  met  for  investment  funding  decisions 
where  exceptions  are  made,  and 

include  a  mechanism  for  reconciling  differences  between  the  IT 
investment  portfolio  and  the  organization's  enterprise  IT  architecture. 

Prerequisite  1:  Adequate  resources  are  provided  for  executing  the 
portfolio  development  process. 

These  resources  typically  involve 

managerial  time  and  attention  to  focus  on  portfolio  development, 

staff  support  for  carrying  out  activities  within  this  critical  process,  and 

supporting  portfolio  development  tools  and  equipment  to  be  used  by  the 
staff. 

Prerequisite  2:  Board  members  exhibit  core  competencies  in 
portfolio  development. 

Understanding  the  principles  behind  the  portfolio  development  process  is 
critical  to  successfully  executing  this  process.  Thus,  training  for  board 
members  may  be  necessary  to  ensure  that  they  are  familiar  with  the  goals 
of  the  process  and  can  carry  out  their  responsibilities  competently. 

Knowledge  building  and/or  training  may  be  provided  ranging  from 

in-depth  courses  for  new  members  to 
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•  a  mandatory  annual  overview  for  all  board  members  of  the  investment 
process,  current  process  modifications,  and  operational  procedures  for 
investment  selection,  control,  and  evaluation. 

Prerequisite  3:  Individual  IT  investments  have  been  analyzed  and 
their  CBSR  data  have  been  validated. 

The  processes  associated  with  the  Stage  3-Investment  Analysis  critical 
process  must  be  in  place  before  the  processes  in  this  critical  process  can 
be  accomplished. 

Prerequisite  4:  The  organization  has  defined  its  common  portfolio 
categories. 

The  organization  has  defined  the  common  portfolio  categories  that  will  be 
used  across  the  organization  when  each  IT  board  creates  its  portfolio  of  IT 
investments  (if  the  organization  has  more  than  one  board).  The  creation  of 
these  common  categories  (1)  aids  in  the  comparison  of  similar 
investments  across  the  organization  and  (2)  helps  create  a  common  set  of 
definitions  if  the  organization  employs  multiple  boards. 

Common  portfolio  categories  should  enhance  decision-making  during  the 
portfolio  development  process.  As  such,  the  organization  should  use 
categories  that  are  easy  to  understand  and  that  correspond  to  the  type  of 
investment  funding  commonly  pursued  by  the  organization.  For  example, 
the  organization  may  wish  to  define  the  categories  on  the  basis  of  the 
investment  lifecycle  (e.g.,  R&D,  full  scale  development,  O&M),  investment 
cost  (e.g.,  <$1M,  $1-5M,  >$5M),  risk  (e.g.,  high,  medium,  or  low),  or 
functionality  (e.g.,  finance,  human  resources,  or  program).  IT  investments 
would  then  be  categorized  by  their  current  stage  in  their  lifecycle  as  each 
IT  board  creates  its  portfolio. 

The  organization  may  also  want  to  define  a  set  of  thresholds  for  each 
common  portfolio  category.  These  thresholds  should  be  meaningful  to  the 
organization,  useful  when  making  investment  decisions,  and  differentiate 
the  categories  from  one  another.  For  example,  an  organization  using 
functional  categories  could  define  CBSR  thresholds  for  each  category, 
such  as 

•  the  maximum  investment  cost  variances  (e.g.,  both  annually  and  in  total), 

•  the  minimum  benefit  that  a  given  investment  is  expected  to  achieve  (e.g.,  a 
return  on  investment  "hurdle  rate"), 
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•  the  maximum  length  of  time  an  investment  should  take  (e.g.,  the  maximum 
allowable  development  time),  and 

•  a  maximum  risk  assessment  score  derived  using  an  industry  accepted  risk 
evaluation  tool. 

A  smaller  organization  with  relatively  few  investments  may  want  to  use  a 
simple  set  of  portfolio  categories. 

Activities  Activity  1:  Each  IT  investment  board  assigns  investment  proposals 

to  a  portfolio  category. 

The  IT  investments  that  have  successfully  completed  the  Investment 
Analysis  process  are  assigned  to  portfolio  categories.  For  example,  each 
investment  in  its  development  phases  might  be  assigned  to  a  development 
category  (with  its  associated  investment  analysis  material  being  made 
available  to  the  investment  board).  The  board  members  then  compare  all 
development  phase  investments  to  one  another  within  that  category. 

The  key  to  this  activity  is  whether  or  not  an  investment  cleanly  falls  within 
a  given  category.  If  an  investment  does  not  fall  within  a  category,  either 
the  investment  should  (1)  be  modified  for  further  consideration  or  (2)  not 
be  funded.  For  example,  an  R&D  category  may  have  a  schedule  threshold 
that  dictates  that  R&D  investments  must  be  completed  in  less  than  1  year. 
Thus,  if  a  high-risk  R&D  investment  is  scheduled  to  take  3  years  for 
completion,  (1)  it  could  be  sent  back  to  the  investment  team  for 
modularization  into  multiple  1  year  investments  with  defined  deliverables 
and  expected  benefits  that  could,  in  the  future,  be  individually  funded, 

(2)  it  could  be  otherwise  modified  for  future  consideration,  or  (3)  it  should 
not  be  funded.  There  may  be  unique  and  unusual  circumstances  where  an 
investment  that  does  not  fit  within  a  predefined  portfolio  category  is 
deemed  sufficiently  important  by  the  organization's  management  to  be 
funded  anyway.  This  type  of  special  investment  should  receive  an 
extraordinary  level  of  management  involvement,  analysis,  and  monitoring 
during  its  life  cycle. 

(See  Stage  3-Investment  Analysis  for  activities  associated  with  investment 
and  proposal  rank  ordering.) 
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Activity  2:  Each  IT  investment  board  examines  the  mix  of 
proposals  and  investments  across  the  common  portfolio  categories 
and  makes  selections  for  funding. 

Once  the  investments  are  assigned  to  portfolio  categories  and  each 
investment  fits  within  each  category,  the  investment  board  completes  the 
selection  process  by  examining  the  portfolio's  mix  of  investments  and 
making  final  investment  decisions.  Executive  discretion  and  managerial 
judgment  may  be  used  during  this  part  of  the  process. 

To  provide  decisionmakers  with  an  understanding  of  the  relative  costs, 
benefits,  schedules,  and  risks  of  each  investment  and  proposal  compared 
to  the  others,  the  organization  may  use  a  scoring  model  or  decision 
support  tool.  Typically,  such  a  model  or  tool  compares  the  costs,  benefits, 
schedules,  and  risks  of  each  investment  or  proposal  against  the 
organizational  investment  criteria  and  assigns  each  investment  proposal  a 
score.  These  scores  are  then  summed  and  normalized  to  produce  a 
cumulative  score  that  establishes  the  investment  or  proposal's  relative 
worth  and  allows  comparison  against  all  of  the  other  investments, 
proposals  and  investments. 

(See  also  the  Executive  Guide:  Measuring  Performance  and  Demonstrating 
Results  of  Information  Technology  Investments,  (AIMD-98-89,  March  1998) 
for  additional  guidance  on  performance  measurement.) 

The  investment  board  may  have  to  reconcile  imbalances  between  total  IT 
funding  expectations  and  funds  required  for  the  qualified  IT  investments 
within  each  portfolio  category.  For  example,  the  investment  board  may 
find  that  the  funding  requests  for  investments  within  the  O&M  category 
are  higher  than  expected  and  that  the  funding  requests  for  investments 
within  the  R&D  category  are  lower  than  expected.  The  investment  board 
can  address  this  problem  by  (1)  leaving  the  outcome  as  it  is,  (2)  modifying 
the  mix  of  investments,  (3)  modifying  investment-level  funding,  or  (4) 
some  combination  of  these  options. 

The  investment  board  can  also  use  other  applicable  sources  of  information 
when  comparing  investments  and  determining  each  investment's  funding. 
While  the  investment  board  should  strongly  consider  the  organizational 
priorities  created  by  the  selection  criteria,  it  may  also  want  to  take  into 
account 

•  the  qualifications,  abilities,  and  achievements  of  the  investment  team, 
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•  unique  or  significant  links  between  the  investment  and  the  organization's 
mission,  strategies,  and  plans,  or 

•  historical  data,  data  on  similar  investments,  or  their  own  investment 
management  experiences. 

Activity  3:  Each  IT  investment  board  approves  or  modifies  the 
annual  C  BSR  expectations  for  each  of  its  selected  IT  investments. 

The  board  modifies  or  approves  annual  CBSR  expectations  for  each 
investment.  Since  some  investments  may  span  multiple  years  and  many 
organizations  perform  investment  selection  on  an  annual  cycle,  the 
investment  board  needs  to  annually  approve  each  investment's 
expectations  (e.g.,  the  CBSR  performance  expectations  for  an  investment 
to  meet  or  exceed  by  the  end  of  1  year).  These  investment  expectations 
should  also  take  into  account  each  investment's  past  performance. 

Additionally,  these  investment  expectations  will  serve  as  the  basis  for 
future  board  reviews,  control  process  activities,  and  post-implementation 
reviews. 

Activity  4:  A  repository  of  portfolio  development  information  is 
established,  updated,  and  maintained. 

The  organization  creates  a  repository  for  storing  information  (e.g., 
investment  CBSR  expectations  and  portfolio  category  thresholds)  related 
to  the  portfolio  development  process.  This  repository  can  be  a  part  of  a 
larger  IT  investment  management  information  system  or  a  component  of 
the  IT  asset  inventory  and  may  be  centrally  or  decentrally  located  within 
the  organization.  Storing  the  information  facilitates  its  use  as  part  of 
control  process  activities,  during  investment  evaluations,  future  selection 
decision-making,  and  future  training  for  board  members. 

Evidence  1:  Physical  evidence  of  Portfolio  Development  exists. 

Physical  evidence  could  include,  for  example, 

•  evidence  of  managerial  time  and  attention  to  portfolio  development, 

•  staff  support  for  portfolio  development, 

•  portfolio  development  tools  and  equipment, 

•  board  members  knowledge  about  the  portfolio  development,  and 
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•  board-created  performance  expectations  for  each  funded  IT  investment. 

Evidence  2:  Documentary  evidence  of  Portfolio  Development  is 
created  and  maintained. 

Documentary  evidence  could  include,  for  example, 

•  a  written  policy  for  establishing  and  maintaining  the  portfolio 
development  process; 

•  individual  IT  investment  analysis  reports; 

•  individual  IT  investment  CB SR  expectations; 

•  common  portfolio  categories,  associated  CBSR  thresholds,  and  funded 
proposals  and  investments  in  each  category; 

•  the  annual  performance  expectations  for  each  funded  IT  investment;  and 

•  a  repository  of  portfolio  development  activities. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Portfolio  Development. 

Testimonial  evidence  could  include,  for  example, 

•  IT  board  member  interviews  and 

•  working  group/support  staff  interviews. 
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Portfolio  Performance 
Oversight 


This  critical  process  builds  upon  the  Stage  2- IT  Project  Oversight  critical 
process  by  adding  the  elements  of  benefit  measurement  and  risk 
management  to  an  organization's  investment  control  capacity.  Compared 
to  less  mature  organizations,  Stage  3  organizations  will  have  the 
foundation  needed  to  control  the  risks  faced  by  each  investment  and  to 
deliver  benefits  linked  to  mission  performance.  Executive-level  oversight 
of  risk  management  outcomes  and  incremental  benefit  accumulation 
provides  the  organization  with  increased  assurance  that  each  IT 
investment  will  achieve  the  desired  CBSR  results.  Expanding  this  focus  to 
the  entire  portfolio  provides  the  organization  with  longer  term  assurances 
that  the  IT  investment  portfolio  will  deliver  mission  value  at  acceptable 
cost. 

The  investment  board's  role  is  not  to  micromanage  each  investment,  but 
instead  to  ensure  appropriate  executive  level  involvement  and 
participation  in  monitoring  each  investment's  progress  toward  achieving 
CBSR  expectations.  These  investment  (and  portfolio)  expectations  are  the 
baseline  for  periodic  performance  reviews  that  examine  the  costs 
incurred,  the  benefits  attained,  the  current  schedule,  and  the  risks 
mitigated,  eliminated,  or  accepted  to  date.  As  such,  this  critical  process 
does  not  focus  on,  for  example,  the  size  and  attributes  of  the  benefits  for  a 
given  investment.  Benefit  expectations  were  defined  during  the  investment 
selection  processes.  Instead,  this  process  focuses  on  how  the  investment 
board  monitors  and  controls  the  investment  portfolio  to  ensure  that  the 
overall  portfolio  provides  the  maximum  benefits  at  a  desired  cost  and  at 
an  acceptable  level  of  risk.  One  way  the  investment  board  performs  this 
executive  level  involvement  is  by  reviewing  the  adequacy  of  the  risk 
management  reviews  conducted  by  the  investment  board's  working  group. 
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Figure  5.1 1 :  Portfolio  Performance  Oversight 
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T o  ensure  that  each  IT  investment  portfolio  achieves  its  C  BSR 
expectations. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  monitoring  and  controlling  portfolio  performance. 

These  policies  and  procedures  typically  specify  the  following: 

•  The  IT  investment  board  is  responsible  for  managing  and  executing  the 
policies  and  procedures  for  portfolio  performance  review. 

•  Actual  investment  C BSR  data  and  investment  CBSR  expectations  are  used 
as  the  basis  for  reviewing  portfolio  performance.  (See  Stage  3-Portfolio 
Development  for  a  description  of  CBSR  expectation  setting.) 

•  The  predetermined  performance  threshold  that  the  IT  investment  board(s) 
should  use  when  analyzing  actual-versus-expected  IT  investment 
performance.  This  threshold  is  typically  defined  on  the  basis  of  the  CBSR 
measures  (e.g.,  more  than  10  percent  over  expected  cost).  However,  it  can 
include  some  other  significant  organization-specific  factors  (e.g.,  the  scope 
of  an  investment  has  grown  to  reach  mission-critical  importance).  This 
predetermined  threshold  will  be  a  major  factor  in  defining  the  remedial 
action  for  underperforming  investments. 

•  The  project  manager  maintains  information  about  the  current  status  of  the 
investment  and  its  CBSR  performance  outcomes. 

•  Changes  to  the  investment's  expectations  and  commitments  are  made  with 
the  involvement  and  agreement  of  the  stakeholders. 

•  The  scope  and  frequency  of  portfolio  performance  reviews. 

•  Investment  performance  is  reviewed  by  a  working  group  composed  of 

•  the  IT  investment  board, 

•  the  project  manager, 

•  the  executive  sponsor,  and 

•  members  of  the  customer  groups  (e.g.,  business  units). 
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Prerequisites  Prerequisite  1:  Adequate  resources  are  provided  for  monitoring 

and  controlling  the  portfolio's  performance. 

These  resources  typically  include 

•  staff  members  for  managing  information  associated  with  tracking 
investment  performance  and 

•  tools  to  support  the  staff  members'  activities  are  made  available. 

Prerequisite  2:  Annual  CBSR  expectations  are  agreed  upon  for 
each  IT  investment. 

Each  investment  has  had  annual  CBSR  performance  expectations  set  for 
it.  These  expectations  should  be  set  as  part  of  the  selection-related  critical 
processes  in  this  stage.  However,  they  may  arise  from  other  processes  if 
the  selection-related  critical  processes  in  this  stage  have  not  yet  been  fully 
implemented. 

(See  Stage  3-Portfolio  Development  for  a  description  of  portfolio  CBSR 
expectation  setting.) 

Prerequisite  3:  The  IT  investment  board  has  access  to  up-to-date 
actual  and  expected  CBSR  data  in  a  repository. 

A  repository  for  investment  performance  data,  potentially  as  part  of  the  IT 
Asset  Inventory,  has  been  established  to  capture,  organize,  and  maintain 
investment  expectations  and  actual  CBSR  data  obtained  during  investment 
reviews.  For  efficiency  reasons,  the  organizations  may  want  to  combine 
this  repository  with  databases  from  other  critical  processes. 

Activities  Activity  1:  Each  IT  investment  board  monitors  the  performance  of 

each  investment  in  its  portfolio  by  comparing  actual  CBSR  data  to 
expectations. 

The  IT  investment  board  is  responsible  for  monitoring  each  investment's 
CBSR  performance.  The  IT  investment  board  examines  actual  investment 
performance  to  date  with  each  investment's  expectations  using  the 
collected  investment  CBSR  data.  The  board  is  notified  of  and  reviews  any 
differences  between  actual  outcomes  and  expectations.  Guidelines  for 
executing  this  activity  include 

•  using  exception  reporting  techniques  to  better  manage  this  activity, 
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•  conducting  this  review  during  a  formal  project  review  activity, 

•  documenting  annual  and  life  cycle  CBSR  expectations  as  a  basis  for  the 
comparison,  and 

•  using  historical  organizational  performance  data  and  industry  baseline 
data  as  a  basis  for  comparisons. 

(See  Stage  3-Investment  Analysis  for  a  description  of  investment-level 
CBSR  expectation  setting  and  see  Stage  3-Portfolio  Development  for  a 
description  of  portfolio  CBSR  expectation  setting.) 

Activity  2:  Using  established  criteria,  the  IT  investment  board 
identifies  its  investments  that  have  not  met  predetermined  CBSR 
performance  expectations. 

The  board  identifies  underperforming  IT  investments  in  its  portfolio  by 
comparing  each  investment's  recent  actual  CBSR  data  to  its  CBSR 
expectations. 

Organizations  may  also  wish  to  use  a  spreadsheet  or  a  graphic  illustration 
(e.g.,  a  management  "stoplight"  with  green,  yellow,  and  red  identifiers)  to 
summarize  this  investment  data.  A  graphic  illustration  provides  a  simple 
way  for  board  members  to  quickly  understand  the  status  of  each 
investment  and  potential  emerging  problem  areas. 

Some  investments  that  the  board  reviews  may  exceed  CBSR  expectations 
(e.g.,  at  lower  costs,  in  less  time,  and  provide  better  benefits  than 
expected).  For  these  investments,  the  board  may  wish  to  accelerate  an 
investment's  funding  or  schedule,  reallocate  resources  within  the  overall 
portfolio,  or  make  some  other  type  of  adjustment. 

Activity  3:  The  IT  investment  board  and  the  project  manager 
determine  the  root  cause  of  the  poor  performance. 

For  each  investment  that  has  not  met  its  predetermined  CBSR 
performance  expectations,  analysis  is  performed  to  determine  the  root 
cause  of  the  poor  performance.  Depending  on  the  size  and  importance  of 
the  investment  within  the  portfolio  and  the  severity  of  the  deficiency, 
external  reviewers  or  experts  may  be  used  to  analyze  the  investment  and 
deficiencies. 

Beyond  determining  the  cause  of  the  deficiencies,  it  is  important  for  the  IT 
investment  board  and  project  management  to  come  to  some  agreement 
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Evidence  of  Performance 


about  the  cause  of  the  deficiencies.  This  will  ensure  that  workable 
solutions  are  created. 

Activity  4:  The  IT  investment  board  and  the  project  manager 
develop  an  action  plan  designed  to  remedy  the  identified  cause(s) 
of  poor  performance. 

Based  on  the  analysis,  the  involved  parties  agree  on  the  corrective  actions 
to  be  executed  as  part  of  the  action  plan.  Typical  corrective  actions 
include 

•  modifying  the  investment, 

•  resolving  resource  constraints  or  capability  problems  contributing  to 
performance  gaps, 

•  working  only  on  one  part  and  stopping  work  on  the  rest  of  the  investment 
(until  a  milestone  is  reached  or  so  as  not  to  exceed  a  set  period  of  time), 

•  temporarily  stopping  the  investment  to  permit  external  work  to  be 
completed,  or 

•  terminating  the  investment. 

Activity  5:  Corrective  actions  are  initiated  and  outcomes  are 
tracked. 

The  investment  board  is  responsible  for  tracking  the  investment  and 
ensuring  that 

•  corrective  actions  as  described  in  the  action  plan  are  executed  by  the 
project  manager  until  the  desired  outcome  is  achieved  and 

•  if  the  project's  performance  has  been  sufficiently  poor,  an  independent 
review  is  conducted  to  ensure  that  all  corrective  actions  have  achieved  the 
intended  results  and  to  determine  whether  additional  changes  or 
modifications  are  still  needed. 

Evidence  1:  Physical  evidence  of  Portfolio  Performance 
Oversight  exists. 

Physical  evidence  could  include,  for  example, 
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•  each  investment  board  monitoring  the  CBSR  performance  of  each 
investment  in  its  portfolio, 

•  portfolio  monitoring  tools  such  as  a  spreadsheet  or  a  graphic 
communication  device, 

•  each  board  identifying  investments  that  are  significantly  underperforming 
their  CBSR  expectations, 

•  each  board  determining  the  root  cause  of  the  investment's  performance 
discrepancies, 

•  each  board  and  its  project  manager  developing  investment  corrective 
action  plans,  and 

•  each  board  tracking  the  implementation  of  each  investment's  corrective 
action  plan. 

Evidence  2:  Documentary  evidence  of  Portfolio  Performance 
Oversight  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example, 

•  portfolio  monitoring  reports, 

•  a  written  policy  for  managing  portfolio  performance, 

•  up-to-date  investment  CBSR  data, 

•  investment  exception  reports  or  investment  CBSR  gap  analysis  reports, 

•  evidence  of  investment  root  cause  analysis,  and 

•  evidence  of  investment  corrective  action  plan. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Portfolio  Performance  Oversight. 

Testimonial  evidence  could  include,  for  example, 

•  board  member  interviews  and 

•  project  manager  interviews. 
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ITIM  Stage 4:  Improving 
the  Investment  Process 


Maturity  Stages 


Critical  Processes 


Jplnvestment  Process  Benchmarking 
□  IT-Driven  Strategic  Business  Change 


Post-Implementation  Reviews  and  Feedback 

□  Portfolio  Performance  Evaluation  and  Improvement 
Systems  and  Technology  Succession  Management 

^Authority  Alignment  of  IT  Investment  Boards 

□  Portfolio  Selection  Criteria  Definition 

□  Investment  Analysis 

□  Portfolio  Development 

□  Portfolio  Performance  Oversight 


□  IT  Investment  Board  Operation 
MIT  Project  Oversight 

□  IT  Asset  Tracking 

Business  Needs  Identification  for  IT  Projects 

□  Proposal  Selection 

IT  Spending  without  Disciplined  Investment 
Processes 


The  primary  focus  of  Stage  4  is  on  using  process  evaluation  techniques  to 
improve  the  overall  performance  of  an  organization's  IT  portfolio.  In 
addition,  the  critical  processes  associated  with  this  stage  help  the 
organization  manage  the  succession  of  low-value  operating  IT  systems  to 
higher-value,  follow-on  investments.  Thus,  this  stage  comprises  the 
following  three  critical  processes: 

Post-Implementation  Reviews  and  Feedback  is  the  process  for 
conducting  post-implementation  reviews  (PIRs)  to  learn  from  past 
investments  and  initiatives  by  comparing  actual  results  to  estimates. 

Criteria:  IT  Assessment  Guide  ( AIMD-10.1.13),  p.  70-72  (CCA,  PRA,  EO 
13011,  GPRA,  CFO,  OMB  A-130);  OM B  IT  I nvestment  Guide,  p.  12; 
Information  Technology  Investment  (AIM  D-96-64),  p.  66. 

Portfolio  Performance  Evaluation  and  Improvement  is  the  process 
for  evaluating  portfolio  performance  and  using  this  information  to  improve 
both  current  IT  investment  processes  and  future  investment  portfolio 
performance. 
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Criteria:  IT  Assessment  Guide  (AIM  D-10.1.13),  p.  73,  78,  80  (CCA,  GPRA, 
OMB  A-130,  OM  B  A-127,  OMB  A-123). 

Systems  and  T echnology  Succession  Management  is  the  process  for 
analyzing  and  managing  the  succession  of  identified  IT  investments  and 
assets  to  their  higher-value  successors. 

Criteria:  SIM  Executive  Guide  (AIM  D-94-115) ;  Year  2000  Computing  Crisis: 
An  Assessment  Guide  (AIM  D-10.1.14),  p.  10;  Capital  Programming  Guide, 
pp.  54-55. 
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Post-Implementation 
Reviews  and  Feedback 


The  purpose  of  a  post-implementation  review  (PIR)  is  to  evaluate  an 
investment  after  it  has  completed  development  (e.g.,  after  its  transition 
from  the  implementation  phase  to  the  O&M  phase)  in  order  to  validate 
actual  investment  results.  This  review  is  conducted  to  (1)  examine 
differences  between  estimated  versus  actual  investment  costs  and  benefits 
and  possible  ramifications  for  unplanned  funding  needs  in  the  future  and 
(2)  extract  "lessons  learned"  about  the  investment  selection  and  control 
processes  that  can  be  used  as  the  basis  for  management  improvements. 
Similarly,  PIRs  should  be  conducted  for  investment  projects  that  were 
terminated  before  completion  to  readily  identify  potential  management 
and  process  improvements. 

The  timing  of  a  PIR  can  be  problematic  -  a  PIR  conducted  too  soon  after  a 
investment  has  been  implemented  may  fail  to  capture  the  full  benefits  of 
the  new  system.  In  contrast,  the  institutional  knowledge  about  a 
investment  can  be  lost  if  the  PIR  is  conducted  too  late.  As  a  general 
guideline,  PIRs  should  be  conducted  within  a  range  of  6  to  18  months  after 
the  investment  begins  its  operational  phase.  However,  this  guideline 
should  be  adjusted  depending  upon  the  nature  of  the  investment  project 
and  expectations  for  the  timing  of  benefit  realizations  documented  in  the 
project  plans. 
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Figure  5.12:  Post-Implementation  Reviews  and  Feedback 
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Purpose 


Organizational  Commitment 


To  compare  outcomes  of  recently  implemented  investments  to  the 
expectations  for  them  and  develop  a  set  of  lessons  learned  from 
these  reviews. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  conducting  PIRs. 

These  policies  and  procedures  typically  specify 

•  who  conducts  and  participates  in  a  PIR; 

•  type  and  size  of  investments  for  which  a  PIR  is  conducted; 

•  when  it  is  appropriate  to  conduct  a  PIR; 

•  what  information  is  presented  in  a  PIR; 

•  if  and  when  the  standard  PIR  process  can  be  tailored  for  a  specific 
investment  and  the  criteria  and  procedures  for  doing  so; 

•  how  conclusions,  lessons  learned,  and  recommended  management  action 
steps  are  to  be  disseminated  to  executives  and  others; 

•  where  PIR  information  and  documents  are  stored  (electronically  or 
otherwise)  for  later  use;  and 

•  when  a  PIR-like  study  should  be  conducted  for  other  IT-related  initiatives 
(such  as  a  strategic  shift  in  technology). 

PIR  contents  should  generally  include 

•  investment  expectations; 

•  actual  investment  results  (e.g.,  end  user  satisfaction,  technical  capability, 
mission  and  program  impact,  unanticipated  benefits); 

•  environmental  changes  that  impacted  the  investment; 

•  a  review  of  the  assumptions  made  during  the  decision-making  period; 

•  expected  next  steps  for  the  i  nvestment; 

•  general  conclusions  (lessons  learned);  and 

•  recommendations  to  executives. 
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Prerequisites  Prerequisite  1:  Adequate  resources  are  provided  for  conducting 

PIRs. 

These  resources  typically  involve 

•  assigning  a  team  to  prepare  and  conduct  each  PIR  with  one  team  member 
assigned  responsibility  for  leading  the  PIR  and 

•  tools  to  support  each  PIR,  such  as, 

•  investment  documentation  in  an  asset  library, 

•  spreadsheet  programs  and  templates, 

•  investment  planning  and  scheduling  programs,  and 

•  risk  and  benefit  assessment  methods  and  tools. 

In  most  cases,  the  project  team  should  actively  assist  the  PIR  team  in 
conducting  the  PIR. 

Prerequisite  2:  Each  IT  investment  board  ensures  that  individuals 
conducting  PIRs  are  trained. 

The  value  of  the  PIR  will  depend  to  a  large  degree  on  the  credibility  and 
competence  of  the  team  members  conducting  the  study.  Thus,  the  PIR 
team  must  be  objective,  well  trained,  and  experienced  when  they  conduct 
the  PIRs.  Also,  the  team  leader  should  have  past  experience  conducting 
similar  investment  reviews. 

Activities  Activity  1:  An  IT  investment  board  identifies  the  projects  for  which 

a  PIR  will  be  conducted  and  a  PIR  is  initiated  for  each  designated 
investment. 

In  accordance  with  organizational  policy,  an  IT  investment  board  will 
identify  and  designate  the  projects  for  which  a  PIR  will  be  conducted.  One 
or  more  examining  teams  will  then  conduct  the  PIR(s)  on  the  designated 
projects.  The  standard  PIR  process  may  be  tailored  to  the  specific 
investment  being  reviewed. 

Typically,  PIRs  will  be  conducted  by  a  centralized  group  under  the 
direction  of  the  enterprisewide  IT  investment  board.  This  approach 
enhances  the  consistency  of  the  resultant  products  and  ensures  coverage 
of  the  appropriate  projects  per  the  organization's  PIR  policy.  However, 
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there  are  other  acceptable  approaches  and  the  organization  should  employ 
an  approach  that  best  meets  its  needs  as  constrained  by  its  resources. 

Activity  2:  Quantitative  and  qualitative  investment  data  are 
collected,  evaluated  for  reliability,  and  analyzed  during  the  PIRs. 

As  part  of  the  objective  analysis  of  the  investment,  quantitative  PIR  data 
are  collected.  These  data  should  largely  arise  from  the  selection  and 
control  process  activities  previously  conducted.  Specific  types  of 
quantitative  data  can  include 

•  CBSR  expectations  and  actual  outcomes, 

•  updated  CBSR  data  and  explanations  for  changes, 

•  objective  measures  of  business  or  mission  impact  such  as  reduced 
operating  cost  or  reduced  product  cycle  time,  and 

•  measurements  of  improved  technical  capability. 

In  addition  to  quantitative  investment  data,  qualitative  information,  such 
as  the  perspectives  and  insights  from  the  project  participants  and  end 
users,  may  serve  to  validate  or  raise  questions  about  the  quantitative 
information  and  the  existing  investment  management  processes  used  by 
the  organization.  Qualitative  data  can  include 

•  surveys  and  interviews  of  end  users,  customers,  project  management, 
project  staff,  contractors,  and  developers, 

•  project  management  and  staff  interviews,  and 

•  interviews  of  senior  decisionmakers  involved  in  investment  oversight. 

Some  common  techniques  for  performing  analyses  during  a  PIR  can 
include 

•  conducting  trend  analysis  using  historical  investment  data, 

•  conducting  means-end  analysis  to  compare  results  with  known  causal 
factors,  and 

•  performing  force  field  analysis  to  understand  the  effects  of  major 
decisions  that  were  made  on  the  investment. 
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Evidence  of  Performance 


Activity  3:  Lessons  learned  and  improvement  recommendations 
about  the  investment  process  and  the  individual  investment  are 
developed,  captured  in  a  written  product  or  knowledge  base,  and 
distributed  to  decisionmakers. 

Lessons  learned  from  the  PIRs  should  be  used  to  recommend  changes  that 
improve  the  (1)  investment  process  (e.g.,  selection,  control,  or  evaluation) 
and  (2)  the  management  of  individual  investments.  For  example, 
recommendations  may  include  suggested  refinements  of  selection  criteria 
for  the  selection  or  control  tasks. 

Once  the  PIR  is  completed,  it  should  be  compiled,  archived,  and 
distributed  to  affected  parties,  particularly  those  with  decision-making 
authority  who  could  most  benefit  from  the  recommendations  and  lessons 
learned. 

Evidence  1:  Physical  evidence  of  Post-Implementation  Reviews  and 
Feedback  exists. 

Physical  evidence  could  include,  for  example, 
the  training  of  PIR  teams; 
actual  PIR  execution; 

quantitative  investment  data  that  have  been  collected,  validated,  and 
evaluated  for  reliability; 

qualitative  investment  data  that  have  been  collected,  validated,  and 
evaluated  for  reliability;  and 

a  PIR  product  or  knowledge  base. 

Evidence  2:  Documentary  evidence  of  Post-Implementation 
Reviews  and  Feedback  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example, 

the  written  policy  for  conducting  PIRs; 

a  portfolio-based  PIR  schedule; 

quantitative  investment  data,  such  as  CBSR  expectations,  actual  data, 
project  plans  and  objectives,  and  business  or  mission  impact 
measurements; 
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•  qualitative  investment  data,  such  as  records  of  contractor  and  developer 
discussions,  end  user  interviews,  customer  surveys,  project  management 
and  staff  interviews; 

•  evidence  of  investment  process  and  individual  investment  lessons  learned; 

•  resulting  recommendations  for  management  action;  and 

•  PIR  product  or  knowledge  base  reports. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Post-Implementation  Reviews  and  Feedback. 

Testimonial  evidence  could  include,  for  example, 

•  PIR  team  member  interviews, 

•  end  user  and  customer  interviews,  and 

•  contractor  and  developer  interviews. 
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Portfolio  Performance 
Evaluation  and 
Improvement 


Ultimately,  an  organization  needs  to  know  how  well  its  collected  pool  of 
investments  in  information  management  and  technology  are  contributing 
to  improvements  in  mission  performance.  Portfolio  performance 
evaluation  and  improvement  is  the  equivalent  of  a  PI  R  for  the  investment 
portfolio.  This  critical  process  seeks  to  determine  how  well  IT  investments 
are(l)  helping  achieve  the  strategic  needs  of  the  enterprise,  (2)  satisfying 
the  needs  of  individual  units  and  users  with  IT  products  and  services,  and 
(3)  improving  IT  business  performance  for  users  and  the  enterprise  as  a 
whole.  To  determine  these  things,  performance  information  for  an 
organization's  entire  portfolio  of  investments  has  to  be  compiled  and 
analyzed  and  trends  examined. 

Key  input  for  these  reviews  include  PIRs,  the  IT  investment  board's 
experiences,  and  major  investment's  results-to-date  extracted  from  control 
process  activities.  These  data  are  generally  project  or  investment-specific 
and  often  are  not  aggregated  for  general  trend  analysis. 
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Figure  5.13:  Portfolio  Performance  Evaluation  and  Improvement 
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To  assess  and  improve  overall  IT  investment  portfolio  performance 
and  the  investment  management  process. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  evaluating  and  improving  the  performance  of  its 
portfolio(s). 

These  policies  and  procedures  typically  specify  the  following: 

•  each  IT  investment  board  is  responsible  for  managing  a  comprehensive 
portfolio  evaluation  and  improvement  process, 

•  access  to  portfolio  data  is  provided  and  confidential/sensitive  data  are 
appropriately  controlled, 

•  each  portfolio  is  evaluated  at  least  annually  to  assess  its  performance, 

•  a  mechanism  for  assembling  and  aggregating  the  investment  performance 
data, 

•  the  key  measures  and  methods  used  to  assess  portfolio  performance  (e.g., 
a  "balanced  scorecard"  approach), 

•  methods  for  analyzing  the  performance  data, 

•  methods  for  comparing  portfolio  performance  and  portfolio  expectations, 
and 

•  a  mechanism  for  reporting  the  analysis  results. 

Prerequisite  1:  Adequate  resources  are  provided  for  conducting  the 
portfolio  performance  evaluation  and  improvement  process. 

These  resources  can  include 

•  support  staff  for  executing  the  activities  in  this  critical  process, 

•  methods  and  tools  to  aid  the  teams  conducting  the  PIRs,  and 

•  current  and  historical  portfolio  data. 

Prerequisite  2:  Board  members  who  are  responsible  for  evaluating 
and  improving  the  investment  processes  and  investment 
portfolio(s)  exhibit  core  competencies  in  portfolio  performance 
evaluation  and  improvement. 
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These  members  must  be  familiar  with  the  IT  investment  management 
approach.  T raining  for  this  critical  process  may  also  include  familiarizing 
executives  with  economic  and  process  management  analysis  techniques. 

T raining  in  quality  management  analysis  and  tools  may  also  be  helpful. 

Knowledge  building  and/or  training  may  be  provided  ranging  from 

•  in-depth  courses  for  new  members  to 

•  an  annual  overview  for  all  board  members  of  the  investment  process, 
current  process  modifications,  and  operational  procedures  for  investment 
selection,  control,  and  evaluation. 

Activities  Activity  1:  Comprehensive  IT  portfolio  performance  measurement 

data  are  defined  and  collected  using  agreed  upon  methods. 

The  portfolio  of  investments  should  be  evaluated  on  its  ability  to  meet 
strategic  needs  of  the  organization,  provide  general  user  satisfaction  with 
product  and  service  delivery  and  management,  and  deliver  effective  and 
efficient  IT  business  functions  (e.g.,  applications  development, 
infrastructure  availability,  project  performance).  A  combination  of 
quantitative  data  and  supporting  qualitative  information  can  be  used  to 
construct  a  picture  of  the  organization's  overall  IT  portfolio  performance. 
This  can  be  analogous  to  developing  a  balanced  scorecard  for  overall  IT 
investment  performance.  (For  more  information,  see  Executive  Guide: 
Measuring  Performance  and  Delivering  Results  of  information  Technology 
Investments,  GAO/AIM D-97-163,  September  1997.) 

Data  collection  and  information  synthesis  should  focus  on  answering  key 
overall  portfolio  performance  questions,  such  as  the  following: 

•  Is  IT  spending  in  line  with  expectations? 

•  Are  we  consistently  producing  cost-effective  results? 

•  How  well  is  the  overall  portfolio  being  managed? 

•  Are  users  satisfied  with  the  products  and  services  being  delivered? 

•  Are  IT  projects  delivering  their  expected  share  of  process  improvements? 

•  How  well  are  integrated  project  teams  being  used  on  major  investment 
projects? 
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Evidence  of  Performance 


Are  quality  IT  products  and  services  being  delivered  within  general 
industry  standards? 

Are  accepted  methods  and  tools  being  used  on  major  systems  investment 
projects? 

Is  the  IT  infrastructure  providing  reliable  and  needed  support  for  the 
organization? 

Measures  should  be  constructed  to  help  objectively  determine 
performance  outcomes  in  these  types  of  areas.  I  n  addition,  the  results  of 
individual  PIRs  as  well  as  internal  and  external  audits  or  reviews  should 
be  examined.  Other  types  of  analyses,  such  as  total  cost  of  ownership,  can 
also  provide  useful  performance  data  on  specific  IT  portfolio  categories, 
such  as  infrastructure  O&M . 

Activity  2:  Aggregate  performance  data  and  trends  are  analyzed. 

T rend  analysis  and  reports  can  help  provide  evidence  that  the  IT  portfolio 
investments  helped  achieve  expected  improvements  in  operational  or 
service  delivery  effectiveness  and  efficiency.  The  development  of  baseline 
performance  data  is  critical  to  making  this  a  meaningful  exercise. 

Activity  3:  Investment  process  and  portfolio  improvement 
recommendations  are  developed  and  implemented. 

Addressing  problems  or  opportunities  usually  involves 

creating  recommendations  for  the  IT  investment  board; 

documenting  the  decision  criteria,  justification,  and  rationale; 

defining  the  expected  benefits  of  the  recommendation; 

making  a  decision  on  implementing  each  recommendation;  and 

tracking  the  recommendation  during  implementation. 

Evidence  1:  Physical  evidence  of  Portfolio  Performance  Evaluation 
and  Improvement  exists. 

Physical  evidence  could  include,  for  example, 

annual  overview  of  the  investment  process, 
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•  collected  portfolio  performance  data, 

•  staff  analysis  of  aggregate  portfolio  data  trends,  and 

•  efforts  toward  investment  process  and  portfolio  improvement 
recommendations  being  developed  and  implemented. 

Evidence  2:  Documentary  evidence  of  Portfolio  Performance 
Evaluation  and  Improvement  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example, 

•  a  written  policy  for  evaluating  and  improving  its  IT  portfolio(s); 

•  portfolio  performance  measurement  approach  and  method  (including 
measures  to  be  used); 

•  portfolio  performance  data,  such  as  cost-effectiveness,  improvements  in 
user  satisfaction,  and  investment  performance;  and 

•  portfolio  trend  analysis  reports. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Portfolio  Performance  Evaluation  and  Improvement. 

Testimonial  evidence  could  include,  for  example, 

•  board  member  interviews  and 

•  working  group  interviews. 


Page  117 


GAO/AIM D-10.1.23  ITIM  Framework  (Version  1) 


Section  5:  Critical  Processes  For  The  ITIM  Stages 
□  ITIM  Stage  4:  Improving  the  Investment  Process 
ODD  Systems  and  Technology  Succession  Management 


Systems  and  Technology 
Succession  Management 


This  critical  process  develops  the  capability  for  (1)  planning  and  managing 
the  migration  of  IT  investments  to  their  successors  (i.e.,  replacement 
systems,  software  applications,  and  hardware)  and  (2)  retiring  low-value 
or  high-cost  IT  investments.  Also,  this  critical  process  enhances  the 
organization's  ability  to  forecast,  plan,  and  manage  the  migration  to  new 
system  investments. 

This  critical  process  is  significant  because  some  IT  investments  can 
outlive  their  usefulness  and  yet  acquire  organizational  inertia  or 
entrenchment,  consuming  resources  that  begin  to  outweigh  their  benefits 
while  obscuring  the  full  cost  of  operations  and  maintenance.  This  inertia 
or  entrenchment  can  often  occur  because  these  assets  (1)  have  created 
important  constituencies  within  the  organization,  (2)  have  a  number  of 
popular  user  features  even  though  the  total  system  cost  exceeds  the  total 
system  benefits,  or  (3)  have  not  had  an  alternative  IT  analysis  performed 
for  it.  The  organizations  at  this  maturity  stage  develop  investment  "exit 
criteria"  such  that  investments  can  be  "de-selected"  appropriately.  The 
critical  process  supports  a  migration  to  a  forward-looking,  solution- 
oriented  view  of  IT  investments. 
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Figure  5.14:  Systems  and  Technology  Succession  Management 
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Purpose 


Organizational  Commitment 


Prerequisites 


T o  ensure  that  IT  investments  in  operation  are  periodically 
evaluated  and  determine  whether  they  should  be  retained, 
modified,  replaced,  or  otherwise  disposed. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  managing  the  IT  succession  process. 

The  organization  has  written  policies  and  procedures  that  define  how  IT 
investments  are  identified,  evaluated,  and  selected  for  succession.  These 
policies  and  procedures  typically  specify 

that  each  IT  investment  board  is  the  cognizant  authority  for  making  IT 
succession  decisions  for  investments  within  its  domain, 

that  the  enterprisewide  IT  investment  board  is  the  cognizant  authority  for 
making  final  IT  succession  decisions, 

the  coordination  of  succession  decisions  across  multiple  IT  investment 
boards, 

the  procedures  for  managing  the  migration  of  IT  systems  to  their 
successors,  and 

the  procedures  for  disposing  of  retired  IT  systems. 

Commitment  2:  An  official  is  designated  to  manage  the  IT 
succession  process. 

An  official  is  designated  to  manage  this  process.  While  the  IT  investment 
board  decides  which  investments  to  continue,  change,  replace,  or  retire, 
this  official  is  responsible  for  managing  the  succession  process  and 
ensuring  that  the  board's  plans  are  executed. 

Prerequisite  1:  Adequate  resources  are  provided  for  conducting  IT 
succession  activities. 

These  resources  typically  involve 

the  attention  of  executives  involved  in  this  process, 

staff  to  support  this  process,  and 

supporting  tools  and  equipment  for  the  staff  to  use. 
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Prerequisite  2:  Investment  board  members  exhibit  core 
competencies  in  IT  succession  decisional  activities. 

To  make  competent  succession  decisions,  board  members  must  have 
sufficient  training  to  carry  out  their  role.  Since  this  critical  process  is 
similar  in  its  core  concepts  to  the  project  selection  process,  the  IT 
succession  training  can  be  tied  to  selection-related  training. 

Knowledge  building  and/or  training  may  be  provided  ranging  from 

•  in-depth  courses  for  new  members;  to 

•  an  annual  overview  for  all  board  members  of  the  investment  process, 
current  process  modifications,  and  operational  procedures  for  investment 
selection,  control,  and  evaluation. 

Prerequisite  3:  Information  from  the  IT  asset  inventory  is  used  by 
the  IT  investment  board. 

The  asset  inventory  is  necessary  to  ensure  that  each  board  is  aware  of  all 
of  the  investments  and  resources  for  which  it  is  responsible  and  to  be 
aware  of  the  cognizant  system  owner/manager(s)  affected  by  succession 
decisions. 

(See  also  Stage  2-IT  Asset  T racking  for  a  description  of  the  activities 
associated  with  developing  an  IT  asset  inventory.) 

Activities  Activity  1:  The  IT  investment  board  develops  criteria  for 

identifying  IT  investments  that  may  meet  succession  status. 

Each  IT  investment  board  develops  the  criteria  that  determine  which  types 
of  investments  are  candidates  for  succession.  In  an  organization  with 
multiple  boards,  the  enterprisewide  board  should  formulate  the  criteria 
first.  The  criteria  should  then  cascade  down  to  the  lower  boards.  A  lower 
level  board  may  have  separate  criteria  for  investments  strictly  within  its 
domain. 

These  candidate  criteria  might  include  investments 

•  at,  near,  or  exceeding  their  planned  life  cycles; 

•  in  their  O&M  phases; 

•  which  have  encountered  significant  data  conversion  problems; 
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•  which  are  based  significantly  on  assumptions  that  are  no  longer  valid  (e.g., 
investments  that  were  based  on  a  type  of  technology  that  is  now  obsolete); 
and 

•  for  which  a  replacement  application  or  hardware  technology  is  imminent 
or  planned. 

(See  also  Stage  3-Authority  Alignment  of  IT  Investment  Boards  for  a 
description  of  the  manner  in  which  multiple  investment  boards  interact.) 

Activity  2:  IT  investments  are  periodically  analyzed  for  succession 
and  appropriate  investments  are  identified  as  succession 
candidates. 

The  defined  criteria  are  applied  to  the  IT  portfolio  to  identify  the 
succession  candidates.  The  analysis  will  generally  be  done  case-by-case, 
looking  at  the  continuing  business  case  and  mission  benefits  surrounding 
each  candidate  and  the  emerging  technologies  as  successor  investments. 
The  analysis  should  be  based  on  the  CBSR  factors  for  each  candidate 
under  consideration  (e.g.,  the  ongoing  costs  of  O&M,  the  risk  of  hardware 
loss  due  to  unavailability  of  spare  parts).  This  analysis  may  require 
managerial  judgment  to  determine  the  merits  of  each  particular  case  or  the 
prospects  for  a  particular  candidate.  Also,  it  is  imperative  that  the 
investment  sponsor,  manager,  and/or  owner  be  involved  with  this  activity. 

Beyond  the  normal  process  of  retiring  older  systems,  this  activity  may  be 
triggered  by  a  variety  of  other  events.  For  example,  after  undergoing  a 
significant  strategic  realignment  or  shift  in  its  underlying  IT  architecture, 
the  organization  will  probably  want  to  engage  in  this  activity  to  ensure  that 
its  IT  resources  are  being  utilized  efficiently. 

(See  also  Stage  2-IT  Asset  T racking  for  a  description  of  the  activities 
associated  with  creating  an  inventory.) 

Activity  3:  The  interdependency  of  each  investment  with  other 
investments  in  the  IT  portfolio  is  analyzed. 

Some  of  the  investments  that  are  identified  as  succession  candidates  may 
be  interdependent  on  other  investments  and  projects.  The  purpose  of  this 
activity  is  to  identify  potential  investment  interdependencies  and  analyze 
the  effects  and  severity  of  succession.  Potential  solutions  to  these 
interdependencies  and  secondary  effects  should  also  be  devised.  The 
board  may  find  it  necessary  to  revise  the  succession  plans  of  some 
investments  based  on  the  analysis  of  effects  on  secondary  investments. 
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Evidence  of  Performance 


Activity  4:  The  IT  investment  board  makes  a  succession  decision 
for  each  candidate  IT  investment. 

Succession  decisions  will  usually  fall  into  the  following  categories: 

Retain/continue  -  Take  no  succession  actions  and  continue  to  operate  and 
maintain  the  current  investment. 

Fix  -  Propose  repairs  to  the  investment  so  that  it  once  again  meets  a 
predefined  level  of  performance  or  business  need. 

Enhance/improve  -  Propose  modifications  to  the  investment  so  that  it 
provides  greater  functionality,  lasts  longer,  or  costs  less. 

Replace  -  Propose  replacing  the  investment  with  a  new  or  different 
investment. 

Combine  or  disaggregate  -  Propose  combining  the  functionality  or 
technical  attributes  of  one  or  more  investments  or  break  the  investment 
apart  into  pieces  and  manage  each  piece  individually. 

Retire/dispose  -  Terminate  the  investment  and  dispose  of  it. 

Succession  plans  are  implemented  as  needed  to  ensure  timely  and 
effective  investment  succession  within  the  context  of  the  overall  IT 
investment  management  process. 

Evidence  1:  Physical  evidence  of  Systems  and  Technology 
Succession  Management  exists. 

Physical  evidence  could  include,  for  example, 

board  members  who  are  involved  with  IT  succession  issues, 

the  existence  of  an  IT  succession  manager, 

the  identification  of  a  pool  of  IT  succession  candidates, 

CBSR  analysis  of  each  succession  candidate  including  candidate's  future 
status  and  system  interdependency,  and 

the  succession  decision  for  selected  IT  candidates. 

Evidence  2:  Documentary  evidence  of  Systems  and  T echnology 
Succession  Management  is  created  and  maintained. 
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Documentary  evidence  could  include,  for  example, 

•  a  written  policy  for  Succession  Management; 

•  IT  succession  criteria,  such  as  life  cycle  milestones,  O&M  phases, 
significant  data  or  processing  conversion  problems; 

•  a  list  of  IT  investment  succession  candidates; 

•  IT  investment  succession  candidate  analysis  reports,  including  CBSR, 
future  status,  and  system  interdependency  analysis;  and 

•  an  IT  succession  decision  list. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Systems  and  T echnology  Succession  Management. 

Testimonial  evidence  could  include,  for  example, 

•  board  member  interviews  and 

•  IT  succession  manager  interviews. 
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ITIM  Stage  5: 
Leveraging  Information 
Technology  for 
Strategic  Outcomes 


Maturity  Stages 


Critical  Processes 


□  Investment  Process  Benchmarking 

□  IT-Driven  Strategic  Business  Change 


ll  Post-Implementation  Reviews  and  Feedback 
^Portfolio  Performance  Evaluation  and  Improvement 
Systems  and  Technology  Succession  Management 

Authority  Alignment  of  IT  Investment  Boards 
^Portfolio  Selection  Criteria  Definition 

□  Investment  Analysis 

□  Portfolio  Development 

□  Portfolio  Performance  Oversight 


□  IT  Investment  Board  Operation 
MIT  Project  Oversight 

□  IT  Asset  Tracking 

Business  Needs  Identification  for  IT  Projects 

□  Proposal  Selection 

IT  Spending  without  Disciplined  Investment 
Processes 


At  Stage  5,  an  organization  leverages  its  IT  investment  capabilities  to  both 
anticipate  the  effects  of  next-generation  information  technologies  and  to 
significantly  drive  strategic  business  transformation.  As  organizations 
harness  the  capability  to  run  effective  management  processes  for 
constantly  selecting,  controlling,  and  evaluating  IT  investment,  they  can 
more  effectively  examine  how  best  to  achieve  major  business 
transformations  to  better  achieve  their  mission.  These  transformations  no 
doubt  will  include  fundamental  changes  made  possible  through  the 
application  of  new  information  technologies  to  support  major  innovation 
in  customer  interaction,  service  delivery  mechanisms,  and  more  effective 
knowledge  management.  One  essential  success  factor  is  to  institute 
effective  processes  capable  of  analytically  sorting  through  more 
technology  choices  of  increasing  complexity. 

Organizations  at  Stage  5  are  focused  on  continuous  improvement  and 
strategic  decision-making  aimed  at  anticipating  and  utilizing  technology 
options  to  drive  desired  business  transformation  outcomes.  Two  critical 
processes  are  central  to  this  stage. 
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•  I  nvestment  Process  Benchmarking  is  the  process  used  to  exploit  IT 
decision-making  to  improve  the  value  of  an  IT  investment  management 
process.  Best  practices  of  other  organizations  are  captured  to  improve  the 
IT  investment  process-leading  to  world-class  outcomes.  The  focus  of 
these  activities  is  cross-functional,  broad,  and  strategic  in  nature. 

Criteria: CCA,  Section  5123  (5);  Benchmarking  course  material  from  CCI, 
Inc.;  Best  Practices  in  information  Technology:  How  Companies  Get  the 
Most  Value  From  Exploring  Their  Digital  Investment,  J  ames  Cortada;  The 
information  Paradox:  Realizing  the  Business  Benefits  of  information 
Technology,  John  Thorpe;  Business  Process  Improvement:  The 
Breakthrough  Strategy  for  Total  Quality,  Productivity,  and 
Competitiveness,  H.  James  Harrington;  Better  Change:  Best  Practices  for 
Transforming  Your  Organization,  PricewaterhouseCoopers. 

•  IT-Driven  Strategic  Business  Change  is  the  process  for  using 
information  technology  to  strategically  renovate  and  transform  work 
processes  and  push  the  organization  to  explore  new  and  better  ways  to 
execute  its  mission. 

Criteria:  CCA,  Section  5123  (5);  Breakthrough  Process  Redesign:  New 
Pathways  to  Building  Customer  Value,  Charlene  Adair  and  Bruce  Murray; 
Transforming  the  Public  Sector,  David  Osborne  and  Ted  Gaebler;  The 
Innovator's  Dilemma,  Clayton  M.  Christensen;  Quality  is  Free:  The  Art  of 
Making  Quality  Certain,  Philip  B.  Crosby. 
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Investment  Process 
Benchmarking 


The  purpose  of  this  critical  process  is  to  measurably  improve  IT 
investment  processes  by  learning  from  and  adopting  the  tools,  techniques, 
or  methods  used  by  best-in-class  external  organizations.  Improvements 
can  include  using  innovative  investment  oversight  tools  and  techniques, 
changing  the  mechanics  of  investment  management,  or  improving  the 
"lessons  learned"  feedback  mechanism.  This  process  is  part  of  an  effort  to 
continually  improve  the  value  of  the  organization's  IT  investments. 

Aspects  of  this  process,  such  as  measurement  of  the  IT  investment 
management  process,  can  be  implemented  in  earlier  stages;  at  Stage  5, 
process  measurement  becomes  an  absolute  necessity. 

Process-based  benchmarking-the  first  step  in  this  critical  process-is  a 
structured  technique  for  measuring  an  organization's  IT  investment 
management  processes.  It  is  different  from  traditional  measurement-based 
benchmarking  where  an  organization  compares  its  performance,  cost,  and 
cycle  time  to  competitors,  industry  averages,  or  a  consultant's  proprietary 
data.  Once  benchmarked,  an  organization's  IT  investment  management 
processes  can  be  modified  and  improved  using  the  tools,  techniques,  or 
methods  learned  from  "best-in-class"  organizations.  The  performance 
gains  resulting  from  implementing  these  process  modifications  can  be 
measured  and  should  result  in  IT  investment  management  processes  that 
meet  or  exceed  the  "best-in-class"  organizations. 
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Figure  5.15:  Investment  Process  Benchmarking 
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□  ITIM  Stage  5:  Leveraging  Information  Technology  for  Strategic  Outcomes 
□□□  Investment  Process  Benchmarking 


To  identify  and  implement  measurable  improvements  in  the  IT 
investment  management  processes  so  that  the  processes  meet  or 
exceed  those  used  by  best-in-class  organizations. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  improving  its  IT  investment  management  process 
using  benchmarking. 

These  policies  and  procedures  typically  specify  the  following: 

•  As  part  of  the  benchmarking  activity,  IT  investment  management  process 
performance  measurements  are  collected  and  analyzed  to  form  a  process 
baseline.  The  investment  management  process  baseline  should  include 

•  the  current  documented  IT  investment  management  process, 

•  performance  measurement  definitions,  and 

•  the  expected  performance  measurement  range. 

•  H  istorical  data  should  be  used  to  analyze  current  performance. 

•  External  organizations  are  evaluated  to  identify  potential  process 
improvement  opportunities. 

•  Significant  changes  to  business  processes  are  approved  by  senior 
management. 

•  The  baselines  and  benchmarks  are  revisited  and  updated  periodically. 

Commitment  2:  An  official  is  designated  to  manage  the 
benchmarking  activities. 

The  organization  designates  an  official  to  manage  this  process.  This 
official  is  responsible  for  managing  the  benchmarking  activities,  ensuring 
that  team  members  are  well  trained,  and  serving  as  the  focal  point  for  this 
critical  process. 

Prerequisite  1:  Adequate  resources  are  provided  for  conducting 
process  benchmarking  activities. 

These  resources  can  include  the  following: 
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•  Individuals  who  are  responsible  for  measuring  investment  process 
performance.  External  organizations  may  also  be  used  to  perform  this 
measurement. 

•  Tools  to  support  investment  process  measurement  are  made  available. 

Prerequisite  2:  Organizational  managers  and  staff  with 
responsibilities  in  this  area  are  trained  in  process  benchmarking 
techniques  or  are  experienced  in  using  these  techniques. 

F or  the  benchmarking  results  to  be  valuable  and  useful,  benchmarking 
team  members  must  know  how  to  conduct  benchmarking  studies.  To 
ensure  benchmarking  competency,  team  members  must  either  receive 
training  or  have  recent  benchmarking  expertise. 

Activities  Activity  1:  Baseline  data  are  collected  for  the  organization's 

current  IT  investment  management  processes. 

The  study  team  measures  the  current  state  of  the  investment  management 
process  to  provide  a  baseline  for  evaluating  expected  and  actual  process 
changes.  Creating  this  baseline  usually  involves  identifying  and  gathering 
process  data  on  the  investment  management  process  components.  These 
data  typically  include 

•  the  level  of  resources  an  organization  expends  conducting  IT  investment 
activities, 

•  quantitative  process  results  such  as  returns  on  investment  and  tangible 
benefits  achieved, 

•  qualitative  process  results  such  as  measures  of  customer  satisfaction  and 
mission  achievement  contributions,  and 

•  the  predefined  range  of  expected  performance  measurement  values. 

Activity  2:  External  comparable  best-in-class  IT  investment 
management  processes  are  identified  and  benchmarked. 

The  purpose  of  this  activity  is  to  find  and  learn  from  organizations  with 
more  efficient  and  effective  investment  management  processes.  Tasks  for 
doing  this  include 

•  identifying  best-in-class  organizations; 
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Evidence  of  Performance 


collecting  data  from  internal,  private,  and  public  sources  about  best-in- 
class  organizations; 

visiting  several  best-in-class  organizations; 

developing  working  relationships  with  one  or  more  of  these  organizations; 
and 

benchmarking  the  best-in-class  organization's  investment  management 
process  components. 

Activity  3:  Improvements  are  made  to  the  organization's 
investment  management  processes. 

Once  an  organization  has  learned  from  the  best-in-class  external 
organizations,  it  must  apply  this  knowledge  to  its  own  processes.  Thus,  the 
organization  should 

decide  on  improvement  goals  and  expectations, 

develop  appropriate  measurable  process  improvement  target  activities, 
and 

analyze,  rank,  and  choose  process  improvement  activities. 

The  organization  then  creates  and  executes  an  improvement  action  plan. 
This  plan  will  vary  with  the  type  and  scope  of  the  benchmarking  studies. 
Executives  should  review  and  approve  the  action  plan  before 
implementing  it  so  that  (1)  they  are  aware  of  the  process  changes  and  (2) 
other  parties  who  may  be  interested  in  the  research  and  process  changes 
can  learn  from  these  initiatives. 

Evidence  1:  Physical  evidence  of  Investment  Process  Benchmarking 
exists. 

Physical  evidence  could  include,  for  example, 

the  identification  of  an  investment  process  benchmarking  manager; 

the  identification  of  a  benchmarking  team; 

the  identification  of  and  baselining  of  the  organization's  IT  investment 
management  processes; 
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•  the  identification,  selection,  and  benchmarking  of  external  best-in-class 
investment  management  processes;  and 

•  the  development  of  benchmark-based  improvement  action  plans. 

Evidence  2:  Documentary  evidence  of  investment  Process 
Benchmarking  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example, 

•  a  written  benchmarking  policy, 

•  the  organization's  IT  investment  management  process  baselines, 

•  IT  investment  management  process  performance  data, 

•  benchmarks  of  external  best-in-class  investment  processes, 

•  benchmark  data  analysis  reports,  and 

•  improvement  action  plans. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  Investment  Process  Benchmarking. 

Testimonial  evidence  could  include,  for  example, 

•  investment  process  benchmarking  manager  interviews  and 

•  benchmarking  team  interviews. 
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IT-Driven  Strategic 
Business  Change 


In  the  previous  ITIM  maturity  stages,  the  organization  invested  in 
information  technologies,  making  certain  that  a  good  business  case  had 
been  defined  within  the  context  of  the  IT  investment  management  process 
and  its  enterprisewide  investment  portfolio.  In  this  maturity  stage,  the 
organization  evolves  its  investment  thinking  toward  managing  IT-driven 
change  of  the  overall  business  process.  IT  can  provide  the  opportunity  to 
change  business  processes  and  leverage  the  organization's  human  capital. 

Information  technologies  can  also  provide  opportunities  for  an 
organization  to  move  dramatically  in  new  directions  in  order  to  meet  its 
goals. 

Citizens  and  countries  are  using  widely  available  computer  encryption 
tools  to  secure  their  communication.  These  tools  can  be  used  for  creating 
"digital  signatures"  which  support  legally  binding  electronic  transactions 
and  help  prevent  fraud. 

The  I  nternet  has  created  opportunities  for  ( 1)  organizations  to  "move 
closer"  to  their  customers;  (2)  business  partners  to  reduce,  if  not  eliminate 
the  need  for  a  third-party  distribution  network;  and  (3)  government 
agencies  to  present  one  common  integrated  service  provider  "face"  for 
service  requests  and  service  delivery  to  the  citizen  (thus  reducing  the  need 
for  local  offices  despite  the  diversity  of  functions  being  executed  at  the 
agency). 

"Smart  munitions"  that  can  find  their  target  in  any  weather,  that  can  be 
reprogrammed  in  flight,  or  that  can  be  controlled  in  real  time  by  a  human 
far  away  from  the  target  are  changing  the  way  war  is  fought  for  some 
components  of  the  military  services. 

Once  an  organization  can  competently  manage  its  IT  investments,  it  must 
proactively  manage  the  potential  of  information  technologies  to 
profoundly  influence  the  strategic  direction  and  outlook  for  the 
organization.  The  organization  must  develop  mechanisms  to  actively  scan 
its  environment  for  new  opportunities  to  utilize  technology.  This  critical 
process  describes  the  activities  associated  with  strategically  employing  IT 
investments  to  change  the  core  processes  of  the  organization. 
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Figure  5.16:  IT-Driven  Strategic  Business  Change 
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Purpose 

Organizational  Commitment 


Prerequisites 


To  dramatically  improve  business  outcomes  by  strategically 
employing  IT  investments. 

Commitment  1:  The  organization  has  written  policies  and 
procedures  for  conducting  IT-driven  strategic  business  change 
activities. 

The  purpose  of  these  policies  and  procedures  is  to  define  the  activities  and 
tasks  to  be  carried  out,  the  roles  of  the  various  parties  when  executing  this 
critical  process,  and  how  these  activities  relate  to  the  organization's 
ongoing  business  activities.  Since  business  managers  may  be  resistant  to 
changing  current  business  processes  based  on  the  promises  of  new 
technology,  these  policies  should  include  incentives  for  management 
participation  in  this  critical  process. 

Commitment  2:  An  official  is  designated  to  manage  the  activities 
within  this  critical  process. 

An  official  is  designated  to  manage  the  creation  and  maintenance  of  IT 
state-of-the-technology  awareness,  identifying  new  information 
technologies,  and  using  selected  technologies  to  plan  and  manage  changes 
to  the  organization's  business  processes. 

Prerequisite  1:  Adequate  resources  are  provided  for  conducting  IT- 
driven  strategic  business  change  activities. 

These  resources  typically  include 

•  funding  support  for  an  IT  state-of-the-technology  laboratory,  test  center,  or 
library; 

•  technical  information  and  research; 

•  funding  for  employing  external  experts  or  reviewers; 

•  staff  support  for  executing  this  critical  process;  and 

•  supporting  tools  and  equipment. 
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Activities  Activity  1:  The  organization  creates  and  maintains  a  knowledge 

base  of  state-of-the-technology  IT  products  and  processes. 

The  organization  creates  the  capacity  to  follow  and  understand  major 
technological  events  and  trends.  This  capacity  can  be  generated  using  one 
of  several  organizational  structures  (e.g.,  an  advanced  technology  group,  a 
cross-departmental  group  of  experts,  a  group  of  external  experts,  or 
technology  centers  of  excellence).  A  designated  official  is  charged  with 
managing  this  group  and  maintaining  the  knowledge  base  and  associated 
IT  awareness  capacity. 

Activity  2:  Information  technologies  with  strategic  business¬ 
changing  capabilities  are  identified  and  evaluated. 

Emerging  trends,  events,  and  technologies  that  have  the  potential  to 
strategically  change  the  organization's  business  are  identified  for  further 
study  (e.g.,  the  growth  of  the  Internet  and  the  World  Wide  Web  or  the 
proliferation  of  wireless  forms  of  communication).  Particular  attention 
should  be  paid  to  breakthrough  technologies  that  have  the  capacity  to 
radically  improve  the  current  working  environment,  business  processes, 
products  or  services,  or  the  organization's  relationship  to  its  customers 
(e.g.,  permitting  staff  to  telecommute  or  to  create  "virtual  communities" 
across  the  Internet).  Also,  to  ensure  that  this  activity  focuses  on  applicable 
information  technologies,  the  organization  should  ensure  that  individuals 
with  business  knowledge  and  experience  are  involved  as  stakeholders  in 
this  activity. 

Activity  3:  Strategic  changes  to  the  business  processes  are  planned 
and  implemented  based  on  the  capabilities  of  identified 
information  technologies. 

Once  a  conclusion  has  been  reached  that  a  set  of  technologies  offers  a 
significant  opportunity,  senior  managers  must  make  the  decision  to  plan 
for  and  engage  in  the  change  to  the  business  processes.  If  the  change  is 
significant  enough,  they  might  wish  to  create  a  separate  organizational 
entity  that  is  (1)  uniquely  positioned  to  take  advantage  of  the  set  of 
technologies  and  (2)  not  beholden  to  the  current  way  of  doing  business. 

As  part  of  planning  these  changes  to  the  business  processes,  the 
organization  should  engage  in  risk-reducing  activities  such  as  pilots, 
simulations,  or  prototypes.  These  risk-reducing  activities  are  particularly 
important  for  large,  complex,  expensive,  or  important  process  change 
initiatives.  The  organization  may  also  want  to  seek  external  review  or 
expertise  when  conducting  these  process  change  activities.  Also,  the 
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organization  should  involve  stakeholders  from  business,  IT  support, 
oversight,  and  customer  groups  when  planning  the  change. 

Evidence  1:  Physical  evidence  of  IT-Driven  Strategic  Business 
Change  exists. 

Physical  evidence  could  include,  for  example 

•  the  rewards  for  supporting  IT-driven  strategic  business  change  activities, 

•  the  identification  of  an  IT  state-of-the-technology  awareness  manager, 

•  the  creation  and  maintenance  of  a  knowledge  base  of  applicable  state-of- 
the-technology  IT  products  and  processes,  and 

•  the  identification  of  applicable  strategic  IT  investments. 

Evidence  2:  Documentary  evidence  of  IT-Driven  Strategic  Business 
Change  is  created  and  maintained. 

Documentary  evidence  could  include,  for  example, 

•  a  written  policy  for  conducting  IT-driven  strategic  business  change 
activities, 

•  documentary  evidence  of  incentives  to  conduct  IT-driven  strategic 
business  change  activities,  and 

•  a  knowledge  base  of  applicable  state-of-the-technology  IT  products  and 
processes. 

Evidence  3:  Testimonial  evidence  is  made  available  during  reviews 
of  IT  -Driven  Strategic  Business  C  hange. 

Testimonial  evidence  could  include,  for  example, 

•  IT  state-of-the-technology  awareness  manager  interviews  and 

•  board  member  interviews. 
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HIM  expands  the  widely  accepted  federal  management  framework  for  IT 
investment  decision-making  embodied  in  OMB  and  GAO  guidance15  and 
shifts  the  content  from  a  guidance-based  focus  to  an  activity-  and  maturity- 
based  focus.  This  shift  reflects  both  the  maturation  of  the  thinking  in  the 
area  of  IT  investment  management  and  the  feedback  we  received  from 
organizations  based  upon  their  experiences  creating  their  IT  investment 
mechanisms  and  processes. 

After  learning  about  several  leading  evaluation  approaches  and  talking 
with  experts  familiar  with  these  approaches,  we  decided  to  develop  a 
maturity-based  framework.  Important  factors  leading  us  to  this  decision 
were  (1)  general  industry  familiarity  and  acceptance  of  maturity-based 
frameworks  (such  as  the  Software  Engineering  Institute's  Capability 
Maturity  Model™)  and  (2)  our  own  working  experience  in  applying 
maturity  model-based  methods  in  IT  audits  and  evaluations. 

After  deciding  on  a  maturity-based  framework,  we  began  to  construct 
ITIM  by  combining  the  content  of  existing  investment  guidance  with  our 
knowledge  of  how  leading  organizations  implement  IT  investment 
management  processes.  To  get  early  feedback  on  this  idea,  we  informally 
presented  our  initial  concept  to  members  of  several  leading  IT 
management  consulting  firms  in  the  Washington,  D.C.,  area  who  are 
engaged  with  federal,  state,  and  private  sector  clients  in  evaluating  or 
designing  IT  investment  management  processes. 

After  receiving  favorable  initial  reviews,  we  proceeded  to  develop  the 
maturity-based  IT  investment  management  framework  by 

•  defining  five  levels  of  IT  investment  maturity  by  stratifying  investment 
management  processes  into  maturity  stages, 

•  identifying  and  documenting  the  processes  critical  for  success  at  each 
maturity  level, 

•  decomposing  each  critical  process  into  key  practices  based  on  a  common 
hierarchy,  and 


15E  valuating  Information  Technology  Investments,  A  Practical  Guide,  Executive  Office  of  the 
President,  Office  of  Management  and  Budget,  November  1995,  and  Assessing  Risks  and 
Returns:  A  Guide  for  Evaluating  Federal  Agencies'  IT  Investment  Decision-making 
(GAO/AIM D-10.1.13,  February  1997). 

SMCapabi I ity  Maturity  Model  is  a  service  mark  of  Carnegie  Mellon  University. 
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•  providing  relevant  examples  and  discussion  for  each  key  practice 
grounded  in  our  research  of  leading  organizations. 

To  ensure  that  ITI  M  would  provide  value  to  the  federal  IT  community,  we 
engaged  in  an  ongoing  review  process  during  the  development  and 
drafting  of  ITI M .  Our  staff  experienced  in  conducting  maturity  model- 
based  assessments  reviewed  initial  drafts.  Later,  we  met  with  and  received 
comments  from  a  selected  group  of  federal  CIOs  and  their  representatives. 
Also,  we  briefed  members  of  the  federal  CIO  Council  and  its  relevant 
subcommittee.  Finally,  we  provided  early  drafts  to  members  of  a  GAO- 
sponsored  advisory  group  of  IT  executives  from  private  industry  and 
federal  and  state  governments. 
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Acquisition:  The  acquiring  by  contract  with  appropriated  funds  of 
supplies  or  services  (including  construction)  by  and  for  the  use  of  the 
federal  government  through  purchase  or  lease,  whether  the  supplies  or 
services  are  already  in  existence  or  must  be  created,  developed, 
demonstrated,  and  evaluated.  Acquisition  begins  at  the  point  when  agency 
needs  are  established  and  includes  the  description  of  requirements  to 
satisfy  agency  needs,  solicitation  and  selection  of  sources,  award  of 
contracts,  contract  financing,  contract  performance,  contract 
administration,  and  those  technical  and  management  functions  directly 
related  to  the  process  of  fulfilling  agency  needs  by  contract. 

Action  Plan:  A  plan  derived  from  recommendations  that  identifies  the 
specific  actions  that  will  betaken  to  improve  a  process  or  a  project  and 
outlines  a  schedule  for  implementing  those  actions. 

Activities:  An  HIM  core  element  that  describes  the  procedures  necessary 
to  implement  a  critical  process.  An  activity  occurs  over  time  and  has 
recognizable  results.  This  core  element  typically  involves  establishing 
plans  and  procedures,  performing  the  work,  tracking  it,  and  taking 
corrective  actions  as  necessary. 

Alignment:  The  degree  of  agreement,  conformance,  and  consistency 
among  organizational  purpose,  vision,  and  values;  structures,  systems,  and 
processes;  and  individual  skills  and  behaviors. 

Assessment:  An  appraisal  by  a  trained  team  of  professionals  to  determine 
the  state  of  an  organization's  current  processes  and  to  determine  the  high 
priority  process-related  issues  facing  an  organization.  An  assessment  may 
also  result  in  organizational  support  for  process  improvement. 

Asset:  Property,  funding,  technical  knowledge,  or  other  valuable  items 
owned  by  the  organization.  Investments  typically  create  assets. 

Benchmarking:  A  structured  approach  for  identifying  the  best  practices 
from  industry  and  government  and  comparing  and  adapting  them  to  the 
organization's  operations.  Such  an  approach  is  aimed  at  identifying  more 
efficient  and  effective  processes  for  achieving  intended  results  based  on 
outstanding  practices  of  other  organizations. 

Benefit:  A  term  used  to  indicate  an  advantage,  profit,  or  gain  attained  by 
an  individual  or  organization.  Tangible  benefits  include  benefits  that  can 
be  explicitly  quantified.  Such  benefits  may  include  reducing  costs, 
increasing  productivity,  decreasing  cycle  time,  or  improving  service 
quality.  Intangible  benefits  include  benefits  that  may  be  easy  to  identify 
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but  that  can  be  difficult  to  quantify.  These  benefits  may  include  more 
efficient  decision-making,  greater  data  accuracy,  improved  data  security, 
reduced  customer  burden,  or  increased  organizational  knowledge. 

Business  Case:  A  structured  method  for  organizing  and  presenting  a 
business  improvement  proposal.  Organizational  decisionmakers  typically 
compare  business  cases  when  deciding  to  expend  resources.  A  business 
case  typically  includes  an  analysis  of  business  process  performance  and 
associated  needs  or  problems,  proposed  alternative  solutions, 
assumptions,  constraints,  and  a  risk-adjusted  cost/benefit  analysis. 

Business  Process:  A  collection  of  related  structured  activities-a  chain  of 
events-that  produce  a  specific  service  or  product  for  a  particular  customer 
or  customers. 

Business  Process  Improvement:  A  systematic  disciplined  approach  that 
critically  examines,  rethinks,  and  redesigns  mission-delivery  processes 
and  sub-processes  within  a  process  management  approach. 

Capability  Maturity  Model™:  A  descriptive  model  of  the  stages  through 
which  organizations  progress  as  they  define,  implement,  evolve,  and 
improve  their  organizational  processes.  This  model  serves  as  a  guide  for 
selecting  process  improvement  strategies  by  facilitating  the  determination 
of  the  current  process  capabilities  and  the  identification  of  issues  most 
critical  to  quality  and  process  improvement. 

Change  Management:  Those  activities  involved  in  (1)  defining  and 
instilling  new  values,  attitudes,  norms,  and  behaviors  within  an 
organization  that  support  new  ways  of  doing  work  and  overcome 
resistance  to  change;  (2)  building  consensus  among  customers  and 
stakeholders  on  specific  changes  designed  to  better  meet  their  needs;  and 
(3)  planning,  testing,  and  implementing  all  aspects  of  the  transition  from 
one  organizational  structure  or  business  process  to  another. 

C  ore  E  lement:  The  five  standard  parts  common  to  each  critical  process 
that  provide  for  its  successful  implementation.  The  five  core  elements  are 
purpose,  prerequisites,  activities,  organizational  commitment,  and 
evidence  of  performance. 

Cost:  A  term  used  to  indicate  the  expenditure  of  funds  for  a  particular 
investment  alternative  over  an  expected  time  period.  Cost  may  include 


5M  Capability  Maturity  Model  is  a  service  mark  of  Carnegie  Mellon  University. 
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direct  and  indirect  initial  costs  plus  any  periodic  or  continuing  costs  for 
operation  and  maintenance. 

Cost/benefit  Analysis:  A  technique  used  to  compare  the  various  costs 
associated  with  an  investment  with  the  benefits  that  it  proposes  to  return. 
Both  tangible  and  intangible  factors  should  be  addressed  and  accounted 
for  in  the  analysis. 

Critical  Process:  A  structured  set  of  key  practices  that,  when  performed 
collectively,  contributes  to  the  attainment  of  a  maturity  stage.  E  ach  critical 
process  is  structured  using  the  five  core  elements. 

Customer:  Individual(s)  or  organizational  entity  for  whom  the  product  or 
service  is  rendered.  The  customer  may  also  be  the  end  user. 

End  User:  The  individual  or  groups  who  will  operate  the  system  for  its 
intended  purpose  when  it  is  deployed. 

Evidence  of  Performance:  An  ITIM  core  element  that  describes  the 
artifacts,  documents,  or  other  proofs  that  support  a  contention  that  the 
key  practices  within  a  critical  process  have  been  or  are  being  executed. 
This  core  element  typically  consists  of  physical,  documentary,  or 
testimonial  evidence. 

Failure:  The  inability  of  a  system  or  component  to  perform  its  required 
functions  within  specified  performance  requirements. 

I  nformation  System:  The  organized  collection,  processing,  transmission, 
and  dissemination  of  information  in  accordance  with  defined  procedures, 
whether  automated  or  manual. 

Information  Technology  (IT):  The  computers,  ancillary  equipment, 
software,  firmware  and  similar  procedures,  services  (including  support 
services),  and  related  resources  used  by  an  organization  to  accomplish  a 
function. 

Institutionalization:  The  building  of  corporate  culture  that  supports 
methods,  practices,  and  procedures  so  that  they  are  the  ongoing  way  of 
doing  business. 

Inventory:  The  organized  and  itemized  list  of  assets  e.g.,  IT  products, 
services,  or  contracts. 
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IT  Architecture:  An  integrated  framework  for  evolving  or  maintaining 
existing  IT  and  acquiring  new  IT  to  achieve  the  organization's  strategic  and 
business  goals.  A  complete  IT  architecture  should  consist  of  both  logical 
and  technical  components.  The  logical  architecture  provides  the  high-level 
description  of  the  agency's  mission,  functional  requirements,  information 
requirements,  system  components,  and  information  flows  among  the 
components.  The  technical  architecture  defines  the  specific  IT  standards 
and  rules  that  will  be  used  to  implement  the  logical  architecture. 

IT  I  nvestment:  The  decision  by  the  organization  to  expend  resources  or 
the  actual  expenditure  of  resources  on  selected  information  technology  or 
IT-related  initiatives  with  the  expectation  that  the  benefits  from  the 
expenditure  exceeds  the  value  of  the  resources  expended. 

IT  I  nvestment  Board:  A  decision-making  body  made  up  of  senior 
program,  financial,  and  information  managers  that  is  responsible  for 
making  decisions  about  IT  projects  and  systems,  based  on  comparisons 
and  trade-offs  between  competing  projects  with  an  emphasis  on  meeting 
mission  goals. 

IT  Investment  Portfolio:  The  combination  of  all  IT  assets,  resources, 
and  investments  owned  or  planned  by  an  organization  in  order  to  achieve 
its  strategic  goals,  objectives,  and  mission. 

IT  Management:  An  approach  used  by  IT  project  managers  to  direct, 
control,  administer,  and  regulate  a  project  team  creating  an  IT  asset  such 
that  the  resultant  product  meets  its  requirements  upon  delivery. 

IT  Project:  An  organizational  initiative  employing  or  producing  IT  or  IT- 
related  assets.  Each  project  has  or  will  incur  costs  for  the  initiative,  has 
expected  or  realized  benefits  arising  from  the  initiative,  has  a  schedule  of 
project  activities  and  deadlines,  and  has  or  will  incur  risks  associated  with 
engaging  in  this  initiative. 

Key  Practices:  The  infrastructures  and  activities  that  contribute  most  to 
the  effective  implementation  and  institutionalization  of  a  critical  process. 

Maintenance:  The  process  of  modifying  a  system  or  component  after 
delivery  to  correct  faults,  improve  performance  or  other  attributes,  or 
adapt  to  a  changed  environment. 

Maturity  Model:  A  model  of  the  stages  through  which  organizations 
progress  as  they  define,  implement,  evolve,  and  improve  their  processes. 
This  model  serves  as  a  guide  for  selecting  process  improvement  strategies 
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by  facilitating  the  determination  of  current  process  capabilities  and 
identification  of  the  issues  most  critical  to  quality  and  process 
improvement. 

Maturity  Stage:  A  well-defined  evolutionary  plateau  toward  achieving 
mature  processes. 

Milestone:  A  scheduled  event  for  which  some  individual  is  accountable.  A 
milestone  is  typically  used  to  measure  progress. 

Mission:  The  enduring,  chartered,  long-term  goal(s)  of  an  organization. 

Modification:  The  act  of  changing  a  system  or  component  to  improve 
performance  or  some  other  attribute  or  to  adapt  the  system  or  component 
to  function  in  a  changed  environment. 

Need:  A  capability  shortfall  such  as  those  documented  in  a  mission  needs 
statement,  deficiency  report,  or  engineering  change  proposal.  A  new 
technology  application  or  breakthrough  may  create  a  new  expressed  need 
by  the  customer. 

Organizational  Commitment:  An  ITIM  core  element  that  describes  the 
management  actions  that  ensure  that  the  critical  process  is  established 
and  will  endure.  This  core  element  typically  involves  establishing 
organizational  policies  and  senior  management  sponsorship. 

Outcome:  The  actual  results,  effects,  or  impacts  of  a  business  initiative, 
program,  or  support  function.  Actual  outcomes  typically  are  compared  to 
expected  outcomes. 

Performance  Measurement:  The  process  of  developing  measurable 
indicators  that  can  be  systematically  tracked  to  assess  progress  made  in 
achieving  predetermined  goals  and  using  such  indicators  to  assess 
progress  in  achieving  these  goals. 

Policy:  A  guiding  principle,  typically  established  by  senior  management, 
that  is  adopted  by  an  organization  to  influence  and  determine  decisions. 

Portfolio:  see  IT  Investment  Portfolio. 

Prerequisites:  An  ITIM  core  element  that  describes  the  conditions  that 
must  exist  within  an  organization  to  successfully  implement  a  critical 
process.  This  core  element  typically  involves  resources,  organizational 
structures,  and  training. 
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Procedure:  A  written  description  of  a  sequence  of  actions  to  be  taken  to 
perform  a  given  task. 

Process:  A  sequence  of  steps  performed  for  a  given  purpose. 

Process  Maturity:  The  extent  to  which  a  specific  process  is  explicitly 
defined,  managed,  measured,  controlled,  and  effective.  Maturity  implies  a 
potential  for  growth  in  capability  and  indicates  the  sophistication  of  an 
organization's  process  and  the  consistency  with  which  it  conducts  these 
processes. 

Project  Manager:  The  individual  with  business  responsibility  for  an 
entire  project.  This  individual  typically  directs,  controls,  administers,  and 
regulates  a  project  developing  or  acquiring  an  information  system. 

Project  Plan:  A  document  that  describes  the  technical  and  management 
approach  to  be  followed  for  a  project.  The  plan  typically  describes  the 
work  to  be  done,  the  resources  required,  the  methods  to  be  used,  the 
procedures  to  be  followed,  the  schedules  to  be  met,  and  the  way  that  the 
project  will  be  organized. 

Project  T earn:  A  group  of  people,  each  with  assigned  responsibilities, 
who  work  closely  together  to  achieve  the  shared  objective  of  delivering, 
operating,  or  maintaining  an  information  system.  The  project  team  may 
work  together  on  tasks  that  are  highly  interdependent  and  may  exercise  a 
level  of  autonomy  in  managing  their  activities  in  pursuit  of  those 
objectives.  The  project  team  may  vary  in  size  from  a  single  individual 
assigned  part-time  to  a  large  organization  assigned  full-time. 

Purpose:  The  desired  outcome  for  each  critical  process. 

Return  on  Investment  (ROI):  A  financial  management  approach  used 
to  explain  how  well  a  project  delivers  benefits  in  relationship  to  its  cost. 
Several  methods  are  commonly  used  to  calculate  a  return  on  investment, 
including:  Economic  Value  Added  (EVA),  Internal  Rate  of  Return  (IRR), 
Net  Present  Value  (NPV),  Payback,  and  the  use  of  nominal  qualitative 
measures. 

Risk:  A  term  used  to  define  the  class  of  factors  which  (1)  have  a 
measurable  probability  of  occurring  during  an  investment's  life  cycle, 

(2)  have  an  associated  cost  or  affect  on  the  investment's  output  or 
outcome  (typically  an  adverse  affect  that  jeopardizes  the  success  of  an 
investment),  and  (3)  have  alternatives  from  which  the  organization  may 
chose. 
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Risk  Management:  An  approach  for  addressing  the  risks  associated  with 
an  investment.  Risk  management  includes  identification,  analysis, 
prioritization,  and  control  of  risks.  Especially  critical  are  those  techniques 
that  help  define  preventive  measures  to  reduce  the  probability  of  these 
factors  from  occurring  and  identify  countermeasures  to  successfully  deal 
with  these  constraints  if  they  develop. 

Schedule:  A  term  used  to  define  the  time  period  corresponding  to  the  life 
of  the  investment.  The  investment  schedule  typically  contains  associated 
phases  and  milestones  that  include:  planning,  proposal  generation, 
acquisition  or  development,  implementation,  operations  and  maintenance, 
and  succession/retirement. 

Selection  C  riteria:  F actors  that  are  identified  for  use  by  an  investment 
review  board  to  identify  and  discriminate  investments  for  subsequent 
funding. 

Stakeholder:  An  individual  or  group  with  an  interest  in  the  success  of  an 
organization  in  delivering  intended  results  and  maintaining  the  viability  of 
the  organization's  products  and  services.  Stakeholders  influence 
programs,  products,  and  services. 

Strategic  Plan:  A  document  used  by  an  organization  to  align  its 
organization  and  budget  structure  with  organizational  priorities,  missions, 
and  objectives. 

Succession  Management:  An  approach  for  determining  when  and  how 
to  replace  current  investments  with  other  investments  that  provide  greater 
benefits  at  lower  costs. 

Threshold:  The  limiting  acceptable  value  of  a  measurement  or  technical 
parameter,  typically  a  performance  requirement. 

Validation:  The  process  of  determining  whether  or  not  the  product 
delivered  at  the  end  of  the  development  process  satisfies  predefined 
requirements. 

Verification:  The  process  of  determining  whether  or  not  the  products  of  a 
given  phase  of  development  fulfill  the  requirements  established  at  the  start 
of  the  phase. 
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This  appendix  describes  the  assessment  process  that  individuals  and 
teams  should  adopt  when  conducting  an  organizational  review  using  the 
Information  Technology  Investment  Management  (ITIM)  framework.  ITIM 
is  a  structured  framework  that  uses  a  growth  and  maturation  approach  to 
assess  an  organization's  IT  investment  management  capability. 

I  n  the  ITI M  framework,  maturity  stages  are  achieved  through 
implementation  of  critical  processes.  These  critical  processes  are  derived 
from  our  research  on  leading  organizations,  our  reviews  of  federal 
agencies,  and  comments  received  from  external  reviewers  during  the  ITIM 
development  process. 

Each  critical  process,  in  turn,  consists  of  five  core  elements  (Purpose, 
Prerequisites,  organizational  Commitment,  Activities,  and  Evidence  of 
Purpose).  The  core  elements  serve  to  define,  establish,  and  institutionalize 
the  critical  process.  The  core  elements  contain  key  practices.  These  key 
practices  are  the  infrastructure  and  activities  that  are  essential  to 
effectively  implementing  and  institutionalizing  a  critical  process. 


Using  ITIM  to  Assess 
Investment  Decision¬ 
making  Processes 


IT  This  assessment  process  guidance  is  designed  to  be  embedded  in  a  formal 
audit  methodology  or  an  organization's  self-assessment  process.  Using  this 
assessment  process,  the  review  team  completes  the  following  phases: 

•  prepares  both  itself  and  the  agency  for  the  assessment, 


•  collects  evidence  of  agency  activities, 

•  evaluates  the  agency  based  on  the  ITIM  framework,  and 


•  briefs  the  agency  on  its  findings. 

Using  this  assessment  approach  facilitates  a  widely  accepted,  repeatable, 
criteria-based  assessment  process  for  auditors  and  agency  managers  when 
conducting  IT  investment  management  assessments.  It  also  provides  the 
organization  with  an  understanding  of  any  investment  management 
process  gaps  identified  during  the  assessment.  However,  before  engaging 
in  an  assessment,  individuals  and  teams  should  do  the  following: 

•  Become  proficient  with  the  ITIM  framework. 

•  Review  the  related  GAO  and  OMB  IT  investment  guidance  (see 
GAO/AI M D-10,1.13;  AIM D-99-32;  AIM D-98-89;  AIM D-94-115  and  OMB 
A-130;  A-ll,  M -97-12;  M-97-02),  U  nderstanding  this  past  guidance  provides 
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greater  insight  into  the  developmental  history,  key  issues,  and  critical 
success  factors  associated  with  the  IT  investment  approach. 

•  Become  familiar  with  generally  accepted  capital  decision-making 
approaches  and  associated  analytical  tools. 

•  Gain  an  understanding  through  training  or  experience  with  the  basic 
concepts  behind  development,  maturation,  and  evolution  of  organizational 
management  skills  and  capabilities  (i.e.,  maturity  models). 

•  Have  experience  assessing  organizations  using  standardized  assessment 
process  and  tools. 


Summary  of  ITIM 
Assessment  Process 


Figure  1 1 1.1  summarizes  the  three  phases  of  the  ITIM  assessment  process. 
E  ach  phase  is  necessary  to  assure  that  the  assessment  team  and 
organization  management  have  sufficient  understanding  of  the  process 
and  the  ITIM  approach,  that  appropriate  evidence  is  collected  to  support 
the  assessment,  and  that  the  conclusions  are  founded  on  the  ITIM 
framework. 
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Figure  111.1:  Phases  in  an  ITIM  Assessment 


ITIM  Assessment  Process 


Phase  1:  Prepare  for 
Assessment 


Present  ITIM  Overview  Briefing  to  the  Organization 

The  assessment  process  begins  with  the  assessment  team  (hereafter 
referred  to  as  the  team)  defining  the  scope  of  the  assessment  (i.e., 
department,  agency,  or  bureau).  The  assessment  scope  will  influence  the 
location  of  the  assessment  (i.e.,  the  physical  place  where  the  majority  of  or 
most  critical  people  and  activities  are  located),  who  the  team  will  brief, 
and  the  extent  of  documentation  required.  Once  the  assessment  scope  is 
defined,  the  team  conducts  an  overview  briefing  for  the  organization  being 
assessed  (hereafter  referred  to  as  the  organization).  This  briefing  covers 
the  ITIM  framework  in  general,  this  assessment  process,  and  any 
organizational-specific  factors  relevant  to  the  job.  The  purpose  of  this 
briefing  is  to  ensure  that  the  organization  understands 

ITIM  and  the  assessment  process  (including  some  techniques  for 
efficiently  and  effectively  performing  an  ITIM  assessment), 
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•  the  anticipated  schedule  of  events, 

•  the  importance  of  involving  the  right  people, 

•  the  general  rules  of  data  collection  and  evidence,  and 

•  the  expected  reporting  process. 

It  is  important  that  the  organization  provides  the  appropriate  personnel  to 
participate  in  the  briefing  and  the  subsequent  HIM  assessment.  The  key 
factor  to  consider  in  ensuring  proper  representation  is  the  role(s)  of  the 
participants  in  the  organization's  IT  investment  activities.  The  following 
participants  will  typically  be  involved: 

•  the  Chief  Information  Officer  (CIO), 

•  the  Deputy  CIO, 

•  representatives  from  the  organization's  IT  investment  board, 

•  representatives  from  the  office  of  the  Chief  Financial  Officer  (CFO), 

•  representatives  from  the  organization's  budget  and  planning  offices,  and 

•  various  IT  managers. 

This  overview  briefing  should  be  sufficiently  early  in  the  assessment 
process  to  allow  the  organization  to  learn  from  the  presentation  and 
prepare  for  the  assessment.  This  typically  means  that  the  briefing  should 
occur  at  least  1  month  before  the  on-site  assessment  activities  begin.  As  a 
result  of  the  briefing,  the  organization  should  be  able  to  expedite  the 
assessment  by  collecting  the  expected  documentation,  identifying  the 
management  processes  for  observation,  and  providing  access  to 
appropriate,  relevant  staff  for  interviews. 

Evidence  of  an  IT  Investment  Management  Process 
A  central  component  of  any  ITIM  assessment  is  the  team's  collection  of 
evidence  about  the  organization's  IT  investment  management  process.  The 
ITIM  framework  guides  the  team's  collection  efforts  by  listing  examples  of 
physical,  documentary,  and  testimonial  evidence  for  each  ITIM  critical 
process.  The  team  should  evaluate  the  variety  of  material  with  respect  to 
the  standards  of  evidence  (sufficient,  competent,  and  relevant)  found  in 
GAO's  Government  Auditing  Standards  ( GAO/OCG-94-4;  also  known  as  the 
"Yellow  Book"). 
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Besides  collecting  documents,  a  typical  ITI M  assessment  may  include 
interviews  with  as  many  as  25  to  50  people  and  numerous  group 
discussions  and  briefings.  Even  more  people  can  participate  through  use 
of  assessment  instruments  such  as  case  studies,  questionnaires,  and 
surveys  (see  the  "Conduct  Case  Study  Reviews"  section  below).  The 
number  of  participants  will  depend  upon  the  assessment  scope  and  the 
organization's  size. 

Obtain  IT  Management  Overview  and  Background  Information 

The  organization  should  provide  the  team  with  one  or  more  information 
briefings  about  the  organization's  overall  IT  investment  management 
process.  It  is  incumbent  upon  the  organization  to  ensure  that  its 
representatives  have  sufficient  knowledge  and  experience  in  managing  IT 
investments  within  the  organization  to  accurately  represent  the 
organization  and  answer  questions.  The  team  should  consider  using  an 
organizational  liaison  for  the  duration  of  the  assessment  to  assist  in 
identifying  and  gaining  access  to  knowledgeable  staff,  providing  access  to 
and  delivering  copies  of  requested  documentation,  and  facilitating  access 
to  physical  evidence. 

The  organization's  overview  briefings  should  provide  a  high-level 
perspective  of  how  the  organization  manages  its  IT  investments.  The 
briefings  are  intended  to  provide  the  team  with  the  following: 

•  An  overview  of  the  organization's  IT  investment  management  process  (i.e., 
what  the  organization  does,  especially  how  it  selects,  controls,  and 
evaluates  its  IT  investments); 

•  An  explanation  of  the  organization's  structure  (who  does  what  as 
documented  in  current  organizational  charts,  especially  any  changes  that 
have  occurred  recently  or  that  are  anticipated); 

•  A  description  of  how  responsibility,  accountability,  and  authority  of  the  IT 
investment  management  process  are  distributed;  and 

•  An  index  of  relevant  documents  (the  IT  investment  management  processes 
contained  in  written  policies,  procedures,  and  guidance,  etc.).  The  index 
should  describe  how  the  organization's  documents  are  laid  out  and  how 
they  relate  to  each  other. 

The  organization  should  also  supply  other  documents  and  background 
information  to  the  team  to  increase  the  team's  efficiency  and  prevent 
misunderstanding  during  the  assessment  process.  The  following 
information  may  also  be  included: 
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•  a  list  of  current  IT  investments  (often  referred  to  as  the  investment 
portfolio); 

•  examples  of  the  data,  information,  and  analyses  upon  which  investment 
decisions  are  based; 

•  descriptions  of  the  decisions  that  are  made  during  the  investment  process; 

•  an  overview  of  the  organization's  mission  and  business  processes  (this 
may  be  contained  in  the  organization's  current  strategic  plan); 

•  terminology  unique  to  the  organization;  and 

•  the  organization's  current  investment  performance  and  process 
improvement  plan. 

Refine  Assessment  Plan 

Based  upon  the  initial  information  it  receives  from  the  organization,  the 
team  may  refine  its  assessment  plan.  For  instance,  the  team  should  reach 
consensus  on  the  critical  processes  and/or  maturity  stages  that  are  not 
applicable.  Specifically,  if  the  organization  has  only  one  IT  investment 
board,  then  the  "Authority  Alignment  of  IT  Investment  Boards"  critical 
process  is  presumed  to  be  rated  as  "not  applicable,"  and  no  further  rating 
is  necessary. 


Phase  2:  Collect  Evidence  Attend  Briefings,  Conduct  Interviews  and  Collect  Documentary 

Evidence 

The  purpose  of  this  set  of  activities  is  to  obtain  supporting  evidence  in 
greater  depth  regarding  the  organization's  implementation  of  the  key 
practices  and  critical  processes  and  to  follow  up  on  issues  or  questions 
arising  from  other  evidentiary  sources  to  date.  The  amount  of  additional 
information  to  be  collected,  and  the  level  within  the  organization  from 
which  it  must  be  obtained,  will  depend  upon  many  factors,  including 

•  the  evidence  obtained  to  date, 


•  the  maturity  of  the  organization's  management  processes, 

•  the  organization's  size  and  complexity,  and 

•  the  scope  of  the  assessment. 
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A  detailed,  revised  data  collection  plan  should  be  developed  based  on  the 
information  required  and  that  received  in  the  initial  overview  and 
background  briefings.  The  team  should  focus  on  the  gaps  that  remain. 

Rather  than  proceeding  sequentially  through  the  critical  processes,  the 
team  may  find  it  more  effective  and  efficient  in  some  situations  to  use 
other  techniques  to  collect  evidence.  These  alternative  techniques  can 
include 

•  collecting  evidence  from  one  organizational  component  at  a  time  for 
multiple  critical  processes  (e.g.,  collect  and  review  all  of  the  IT 
investment-related  policies  from  a  central  policy  review  committee); 

•  collecting  evidence  for  one  single  stage  from  multiple  organizational 
components  (e.g.,  collect  and  review  all  evidence  for  Stage  2  at  onetime); 
or 

•  collecting  evidence  for  one  ITIM  component  across  all  organizational 
components  (e.g.,  collect  and  review  all  evidence  relating  to  organizational 
Commitment). 

If  the  organization  states  that  it  is  implementing  a  critical  process  using 
some  set  of  practices  other  than  the  ones  described  in  ITIM,  then  these 
practices  should  be 

•  clearly  delineated, 

•  formally  approved  by  the  organization,  and 

•  convincingly  supportive  of  the  intent  of  the  critical  process  which  it  is 
supposed  to  supplement. 

The  organization  may  also  provide  for  the  team  an  in-depth  walkthrough 
of  specific  key  practices  within  a  critical  process.  This  provides  the  team 
with  physical  evidence  of  a  critical  process  and  would  also  support  the 
documentary  evidence  associated  with  an  assessment  of  a  critical  process. 

Obtain  Briefings 

Briefings  at  this  point  should  be  focused  on  those  critical  processes  and 
key  practices  which  lack  sufficient  documentation  following  the  initial 
background  briefings.  Processes  and  practices  that  are  known  to  be 
missing  in  the  organization  may  be  skipped.  Presenters  should  be 
encouraged  to  bring  documentation  to  the  briefings  for  distribution.  In 
many  instances  the  briefings  may  actually  evolve  into  discussions  as  the 
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team  focuses  on  the  supporting  evidence  of  existing  investment 
management  processes. 

Conduct  Interviews 

The  purpose  of  these  interviews  is  to  collect  supporting  evidence  from 
agency  officials  and  staff  who  directly  participate  in  the  IT  investment 
management  process  (e.g.,  executives,  managers,  support  personnel). 
Interviewing  a  variety  of  organization  staff  assists  the  team  in  determining 
the  extent  to  which  the  investment  process  policies  and  procedures  have 
been  communicated  throughout  the  organization.  These  interviews  should 
also  point  the  team  to  other  documentary  evidence  (probably  located 
within  investment  projects)  and  guide  the  evidence  collection.  (Also  see 
"Conduct  Case  Study  Reviews"  below.) 

Collect  and  Review  Documentary  Evidence 

The  purpose  of  this  step  is  to  review  the  documentary  evidence  of  how  the 
investment  management  processes  are  actually  implemented,  and  how 
well  the  evidence  correlates  to  the  ITIM  key  practices.  This  activity  is 
repeated  for  each  key  practice  that  is  conducted  within  the  organization. 

The  team  will  typically  begin  by  collecting  broad,  organization-level 
evidence  (e.g.,  policy  planning  documents).  This  evidence  will  lead  the 
team  to  lower-level,  implementation-oriented  documentation  (e.g., 
meeting  notes  and  working  papers).  In  this  process  the  team  will 

•  determine  what  documentary  evidence  is  available  based  on  information 
provided  at  briefings  and  interviews, 

•  request  or  collect  documentary  evidence, 

•  evaluate  the  documentary  evidence,  and 

•  organize  the  evidence  according  to  the  key  practices  within  the  ITIM 
framework. 

Consolidate  Evidence  and  Collect  Follow-up  Evidence 

Before  the  team  can  make  rating  judgments  of  the  key  practices,  core 
elements,  critical  processes,  and  maturity  stages  under  consideration,  they 
must  complete  the  following: 

•  determine  whether  or  not  the  evidence  provides  a  sufficient,  competent, 
and  relevant  basis  for  making  a  rating  judgment; 
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•  assemble,  organize,  and  analyze  the  collected  evidence  and  consolidate  it 
into  a  manageable  summary  of  evidence  according  to  the  IT  I M  framework; 
and 

•  determine  the  follow-up  evidence  required  to  make  a  rating  judgment  and 
a  method  to  collect  this  evidence.  The  team  must  also  decide  how  to 
proceed  if  (1)  there  is  no  other  evidence  available  or  (2)  the  available 
evidence  is  ambiguous  and/or  inadequate. 

Invariably  the  team  will  identify  the  need  for  additional  analyses  or  follow¬ 
up  evidence  to  complete  the  assessment.  The  team  can  either  send  written 
questions,  requests  for  specific  evidence,  or  conduct  follow-up  interviews 
to  collect  this  required  evidence. 

Determine  Evidence  Sufficiency,  Competency,  and  Relevancy 
In  order  to  achieve  accurate  and  reliable  ratings  in  the  assessment 
process,  the  following  evidence  guidelines  must  be  met  while  evaluating 
the  collected  evidence: 

•  There  should  be  sufficient  evidence  collected  from  two  or  more 
(preferably  independent)  sources  to  support  a  rating. 

•  The  evidence  must  be  corroborative  and  directly  relevant  or  logically 
linked  to  the  key  practice  and  critical  process. 

•  The  evidence  must  provide  adequate  coverage  and  be  competent.  M  ore 
specifically, 

•  testimonial  evidence  must  be  from  interviews  with  or  presentations  by 
the  staff  who  perform  the  related  investment  management  process; 

•  original  documentary  evidence  must  be  a  direct  result  of  executing  the 
investment  management  process;  and 

•  physical  observations  must  be  made  by  team  members  or  other 
credible,  unbiased  third  parties. 

Under  some  circumstances,  the  team  may  decide  that  confirmation  from 
three  or  more  separate  evidentiary  sources  is  needed.  For  example,  the 
team  may  realize  that  a  particular  individual's  interview  is  significant 
enough  that  it  may  cause  a  critical  process  to  be  rated  as  "not 
implemented."  In  this  case,  the  team  may  decide  that  this  interview,  as  a 
single  source  of  evidence,  warrants  corroboration  from  other  interviews. 
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As  a  general  rule,  if  there  is  any  doubt  about  whether  a  rating  is  valid,  the 
team  should  initiate  additional  information  collection  efforts. 

Consolidate  Evidence 

Consolidation  helps  the  team  sift  through  and  organize  the  large  quantity 
of  evidence  that  is  typically  acquired  during  an  assessment.  Evidence 
consolidation  also  provides  an  opportunity  for  the  team  to  share 
interpretations  of  the  collected  evidence  and  enables  the  team  members  to 
develop  a  consensus  on  rating. 

During  evidence  consolidation,  the  team  assesses  their  progress  toward 
their  goals  and  reviews  the  evidence  collected  to  that  point  in  time.  While 
no  particular  format  is  mandatory,  these  steps  are  typically  followed 
(often  they  are  repeated  multiple  times): 

•  Team  members  index,  review,  and  assess  the  evidence  collected  to  date. 

•  Team  members  identify  key  practices  that  require  further  clarification. 

•  Team  members  share  opinions  of  the  sufficiency  of  the  evidence  and 
develop  preliminary  ratings  based  on  team  consensus. 

If  the  team  cannot  reach  consensus,  it  must  ( 1)  identify  the  evidence 
needed  to  resolve  the  outstanding  issues  and  (2)  generate  a  plan  for 
collecting  the  needed  evidence. 

Conduct  Case  Study  Reviews 

The  team  may  choose  specific  IT  investment  projects  for  in-depth  reviews 
to  validate  organization-level  evidence  and  to  better  understand  the 
organization's  IT  investment  management  process.  The  decision  of 
whether  to  conduct  case  studies  will  depend  on  whether  additional 
evidence  is  required  to  document  investment  processes.  By  evaluating  the 
actual  investment  processes  used  with  a  variety  of  investment  projects, 
the  team  obtains  a  clearer  picture  of 

•  the  investment  processes  as  they  have  actually  have  been  implemented, 

•  the  consistency  with  which  the  investment  process  is  executed, 

•  evidence  of  whether  the  organization's  policies  and  procedures  have  been 
communicated  to  the  project-level, 

•  the  commitment  that  the  organization  has  to  the  investment  process,  and 
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•  the  beneficial  effects  that  improvements  in  these  processes  might  have  on 
the  performance  of  the  organization. 

Select  Investment  Projects 

The  team  should  select  one  or  more  investment  projects  in  each  major 
lifecycle  phase  (e.g.,  R&D,  full-scale  development,  and  O&M).  At  least  one 
of  the  cases  should  include  a  high-cost  and/or  high-risk  investment  project. 
For  each  project,  the  team  should  follow  the  history  of  the  investment 
project  as  it  has  cycled  through  the  organization's  IT  investment  process. 
Projects  may  be  selected  on  the  basis  of  whether  the  required 
documentation  is  available,  though  this  approach  may  bias  the 
conclusions  drawn  from  the  evidence. 

Select  Participants 

The  team  also  needs  to  determine  whom  they  expect  to  participate  in 
these  project-level  reviews.  In  all  cases,  participants  should  come  from  the 
investment  projects  selected  and  the  organizational  groups  that  support 
those  investment  projects.  It  may  also  be  necessary  to  include  people 
selected  from  other  organizational  components  (e.g.,  IT  investment 
oversight  staff). 

Execute  Review(s) 

These  reviews  will  typically  cover  the  following  dimensions: 

•  Process  conformance-the  degree  to  which  the  project  being  reviewed 
went  through  the  agency's  IT  investment  decision-making  process. 

•  Data  sufficiency,  quality,  and  completeness- the  type,  accuracy,  and  value 
of  the  data  used  to  make  investment  decisions  about  the  project. 

•  Decisions  executed-the  type  of  decision  made  and  the  degree  to  which  it 
was  executed. 

Reconcile  Differences 

I  n  some  situations,  the  results  of  the  case  studies  may  contradict  the 
preliminary  ratings  developed  during  the  assessment  of  the  organization. 

I  n  this  case,  the  team  should  investigate  the  contradiction(s),  determine 
their  root  cause,  and  modify  the  preliminary  rating(s)  if  necessary.  As 
mentioned  before,  the  purpose  of  the  case  studies  or  surveys  is  to  provide 
additional  corroborative  evidence  for  the  organizational  ratings  and 
conclusions  reached  during  the  organization-level  ITIM  assessment. 
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Phase  3:  Determine  Ratings 
and  Finish  Assessment 


Determine  Final  Ratings 

Once  evidence  collection  is  complete,  the  team  must  assess  the 
consolidated  evidence  and  decide  whether  each  key  practice,  core 
element,  critical  process,  and  maturity  stage  has  been  successfully 
executed.  The  team  makes  final  rating  judgments  as  a  group.  Developing  a 
consensus,  so  that  the  majority  agrees  and  no  one  is  opposed,  ensures  that 
the  decision  is  fair  and  that  all  evidence  has  been  considered. 


HIM  is  a  hierarchical  framework,  so  the  rating  of  each  higher-level 
component  is  entirely  dependent  on  the  components  below  it.  That  is,  if 
any  key  practice  is  not  executed  satisfactorily,  its  corresponding  critical 
process  is  not  implemented  satisfactorily,  and  the  corresponding  maturity 
stage  cannot  be  considered  complete.  Because  of  this  hierarchical 
prioritization,  the  team  must  begin  by  rating  key  practices  and  work  their 
way  up  the  hierarchy.  The  sequence  of  ratings  is  as  follows: 

•  key  practices  are  rated  first, 

•  core  elements  are  rated  second, 

•  critical  processes  are  rated  third,  and 

•  the  ITIM  stage  is  determined  last. 

The  team  members  should  devise  a  method  and  mechanism  for  tracking 
and  documenting  the  rating  judgments  as  they  are  being  made.  Besides 
creating  a  reproducible  "audit  trail,"  these  supporting  documents  are 
useful  when  delivering  summary  results. 

Determine  Each  Key  Practice  Rating 
Key  practices  are  rated  as  being 

•  "executed"  or 


•  "not  executed." 

An  ITIM  key  practice  is  successfully  "executed"  if  (1)  the  team  judges  that 
the  key  aspects  of  the  practice  are  being  executed  by  the  organization  or 
(2)  the  organization  presents  the  team  with  convincing  evidence  that  an 
alternative  practice  achieves  the  same  outcome.  An  ITIM  key  practice  is 
"not  executed"  if  there  are  significant  weaknesses  in  the  organization's 
execution  of  the  practice  and  no  adequate  alternative  is  in  place.  If  the 
team  has  found  no  evidence  of  a  practice  during  the  assessment  process, 
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that  result  may  constitute  physical  evidence  of  a  key  practice  "not 
executed"  rating. 

If  the  team  rates  a  key  practice  as  "not  executed,"  the  organization  should 
be  given  an  opportunity  to  produce  evidence  that  might  mitigate  or  refute 
the  evidence  that  indicated  this  rating.  By  double-checking,  the  team 
avoids  making  ratings  based  on  incorrect  information. 

Determine  Each  Core  Element  Rating 
Core  elements  are  rated  as  being 

•  "fulfilled"  or 

•  "not  fulfilled." 

A  core  element  is  successfully  "fulfilled"  if  the  (1)  team  judges  that  each  of 
the  key  practices  within  the  core  element  is  executed  or  (2)  the 
organization  presents  the  team  with  convincing  evidence  that  an 
alternative  approach  achieves  the  same  outcome.  A  core  element  is  "not 
fulfilled"  if  there  are  significant  weaknesses  in  execution  of  any  of  the  key 
practices  within  the  core  element  and  no  adequate  alternative  is  in  place. 

Determine  Each  Critical  Process  Rating 
Critical  processes  are  rated  as  being 

•  "implemented," 

•  "not  applicable," 

•  "not  implemented,"  or 

•  "not  implemented,  but  improvements  underway." 

An  HIM  critical  process  is  "implemented"  if  its  underlying  key  practices 
and  core  elements  are  successfully  implemented  or  if  a  satisfactory 
alternative  is  in  place.  The  HIM  critical  process  "Authority  Alignment  of  IT 
Investment  Boards"  is  "not  applicable"  if  the  organization  has  only  one  IT 
investment  board.  An  ITIM  critical  process  is  "not  implemented"  if  there 
are  significant  weaknesses  in  the  assessed  organization's  implementation 
of  the  underlying  key  practices  and  core  elements  and  no  adequate 
alternative  is  in  place.  An  ITIM  critical  process  is  "not  implemented,  but 
improvements  underway"  if  over  half,  but  not  all,  of  its  underlying  key 
practices  and  core  elements  are  rated  as  being  executed.  For  example,  if 
well-defined  policies  and  procedures  have  been  developed,  but  no  training 
has  been  established,  the  critical  process  would  be  rated  as  "not 


Page  161 


GAO/AI MD-10.1.23  ITIM  Framework  (Version  1) 


Appendix  III 

Guidance  for  Conducting  an  HIM  Assessment 


implemented,  but  improvements  underway."  This  rating  is  intended  to 
indicate  that  the  organization  has  made  progress  in  addressing  the  critical 
process,  but  the  work  has  not  been  completed. 

A  critical  process,  like  key  practices  and  core  elements,  can  be 
implemented  by  alternative  means.  The  crucial  point  to  assessing  an 
alternative  approach  is  that  the  techniques  used  to  fulfill  the  purpose  of 
the  critical  process  must  be  defined,  implemented,  and  institutionalized. 
These  are  the  same  criteria  used  to  assess  the  adequacy  of  an 
organization's  execution  of  a  practice,  core  element  or  critical  process 
described  in  the  ITIM  framework. 

Determine  Investment  Management  Stage 

All  of  the  critical  processes  within  a  particular  investment  management 
maturity  stage,  and  within  each  lower  stage,  must  be  rated  as 
"implemented"  or  "not  applicable"  in  order  for  the  organization  to  achieve 
that  stage  rating.  For  example,  for  an  organization  to  be  rated  as  an  ITIM 
Stage  3  organization,  all  of  the  critical  processes  within  both  Stage  2  and 
Stage  3  must  be  rated  as  being  "implemented"  or  "not  applicable"  by  the 
team. 

Deliver  Draft  Summary  Assessment 

The  final  step  in  the  assessment  process  is  the  delivery  of  draft  results  to 
the  organization.  In  addition,  these  draft  results  can  form  the  basis  for  the 
development  of  a  full  audit  report  if  one  is  requested.  The  draft 
assessment,  typically  in  the  form  of  a  briefing,  contains 

•  an  itemization  of  ITI M  critical  processes  that  have  been  assessed  and 
rated; 

•  an  identification  of  implemented  critical  processes,  an  identification  of  the 
achieved  investment  management  stage,  and  graphical  or  summary 
representations  of  the  above  information; 

•  a  rating  of  each  key  practice  for  each  critical  process  that  was  assessed; 
and 

•  an  evidence-based  rationale  for  each  rating  determination. 

The  team  can  use  the  case  study  reviews  to  illustrate  the  ratings  and 
conclusions  that  the  team  reached  as  a  result  of  the  assessment.  I  n  order 
to  focus  on  the  key  practices  needing  improvement  the  team  typically  will 
deliver  draft  ratings  only  for  key  practices  judged  to  be  "not  executed." 
This  approach  optimizes  time  overall  and  ensures  maximum  time  is  spent 
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corroborating  investment  management  weaknesses  and  collecting 
additional  evidence  about  them  or  other  areas. 
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